acceptodds
Under review as a conference paper at ICLR 2027

Generative Neural Retargeting for Human-to-Robot Dexterous Manipulation

Abstract

Human demonstrations are a scalable data source for learning dexterous manipulation, but the embodiment gap prevents human motion from being executed directly on robots. Inverse kinematics (IK) retargets human motion to robots efficiently but ignores dynamics, often producing infeasible motions. Reinforcement learning (RL) and sampling-based model predictive control (MPC) are commonly employed to yield dynamically feasible motions, but both are sample-inefficient and sensitive to hyperparameters. RL suffers from costly and unstable training and tedious reward engineering; MPC avoids policy optimization, yet retargets each trajectory in isolation, and solving one does not make the next easier. Sampling cost grows rapidly with dataset size and task difficulty. We hypothesize that dynamically feasible trajectories concentrate near a low-dimensional manifold shared across demonstrations, so that retargeting can be reduced to sampling from that manifold, conditioned on human motion, rather than solving a fresh optimization problem for every demonstration. We propose Generative Neural Retargeting (GNR), which learns the manifold of feasible trajectories with a flow matching model and samples from it to efficiently retarget human motions. GNR outperforms MPC with only 8.5% of the samples required by MPC, achieving a success rate of 56.20% compared to 27.20% for MPC. GNR can be used for scalable and efficient retargeting of large-scale and high-precision human demonstrations: by applying GNR within a real-to-sim data engine, we produce a dexterous manipulation dataset with dense contact-force labels, spanning 195k demonstrations and 3.2k object geometries.

Primary area:
applications to robotics, autonomy, planning
Keywords:
robotics, dexterous manipulation, robot learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_xGToOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors proposed to use a generative model to do physics-based dexterous retargeting from human demonstrations, which they call GNR. Specifically, they use a conditional flow matching model to learn the residual correction control sequence delta u conditioned on the reference state trajectory and object geometry. This model is trained on MPC generated data, then proven to work to some degree on cases where MPC fails. It is capable to do test time scaling by sampling from the generative model many times. They developed a data engine using GNR and collected a physics-based dexterous manipulation dataset.

Critical strengths
  1. The authors focus on scaling up dexterous manipulation data, which is an important problem.
Critical weaknesses
  1. It is unclear whether GNR generated data can be beneficial for learning-based algorithms. In figure 17, the curves are too noisy to judge visually which dataset is significantly better. Although the best checkpoint SR shows GNR and GNR+MPC are stronger, the overall trends of the curves are too noisy to make this result convincing. Perhaps it is helpful to re-evaluate figure 17 using a larger number of test episodes to reduce noise and reveal a clearer trend. Also, this experiment should be in the main paper. As a paper that proposes a data generation method and data engine, it is very important to experimentally prove that the generated data (quantity plus quality) is beneficial for downstream learning methods while comparing against existing datasets.
  2. No real world experiments are provided.
  3. Experimental comparisons are only against analytic methods. The authors mentioned RL-based methods but never compared against them experimentally.
Questions for authors
  1. In the supp material coffee_pot folder, the human was grasping the body of the cup, while robot was grasping the handle of the cup, why is there an inconsistency?
  2. Could the authors provide insights into why GNR is able to generalize to OOD objects while only trained on 23 objects?
  3. In the comparisons of efficiency in Table 2, is GNR also evaluated using only CPU? Or is it allowed to use GPU? Does the MPC baseline only use CPU?
  4. What are the differences between OOD object and OOD geometry? The authors claim a dataset spanning 3.2k object geometries in the abstract, could they also provide the span of the number of distinct objects?
3 comments
AuthorsOpenReview ↗

Response to Reviewer xGTo

We thank the reviewer for acknowledging the importance of scaling up dexterous demonstrations and for the constructive suggestions! We added several new experiments and clarifications to address the concerns. Real-robot and simulation demos are also included on the anonymous webpage for review: https://gnr-review.github.io/.


W2 — Real-robot experiments

We added real-robot experiments to Sec. 5.4 based on the suggestion. We are glad to show that a visuomotor policy trained with GNR data achieves zero-shot sim-to-real transfer on a highly challenging dexterous task. To our knowledge, this is also the first result in retargeting studies that achieves millimeter-precision policy sim-to-real transfer over long horizons for five-fingered dexterous manipulation. Dexterous retargeting and sim2real is generally very challenging, and state-of-the-art methods primarily retarget trajectories for tasks with 50–150 mm precision over a few seconds (e.g., CHORD, SPIDER, DexTrack).


W1 — Can GNR-generated data benefit downstream learning?

GNR can benefit downstream learning with matched or improved data quality while reducing dataset generation cost by 4x-11.2x. As suggested by the reviewer, we moved this experiment from the appendix to the main paper Sec. 5.4 alongside our new sim-to-real results. We added 3-seed mean/std results to Fig. 6 to compare policies trained on GNR, MPC, and GNR+MPC datasets. Each checkpoint on the curve is evaluated on 500 held-out episodes. At 200k training steps, policies trained on GNR and GNR+MPC data achieve 53.4% and 61.3% SR, respectively, compared with 55.2% for MPC. GNR yields similar policy success, and *GNR+MPC improves success by 6.1 pp, while GNR reduces data-generation cost by 11.2× and GNR+MPC by about 4×.


W3 — Comparison with RL baselines

We added two RL baselines, DexTrack and ManipTrans, to App. B.2. We adapt both to our simulator following their implementations. GNR uses 256 samples (13k environment steps) in open loop without policy optimization and achieves 56.1% SR. With 3,000 training rollouts (162k steps) per demonstration, DexTrack and ManipTrans reach 53.3% and 51.7%, at 32× and 75× higher CPU cost than GNR.


Q1 — Human and robot video demos in the supplementary material

We revised the video demos in App. K.3 and on the anonymous website so the human video and robot rollouts show the aligned motion.


Q2 — Why does GNR generalize when trained on 23 objects?

Following your suggestion, we added a discussion of GNR's generalization to App. B.4. This generalization benefits from both data and test-time scaling. Because the egocentric videos in HOT3D cover a limited set of objects, our real-to-sim engine broadens data coverage by stretching and deforming object meshes (see Fig. 12) and varying motions and physical parameters. Objects with different overall geometries can share similar local geometries at contact points, so the model can reuse contact patterns learned from the training objects. Test-time scaling complements data scaling by increasing the likelihood of generating a feasible trajectory with more samples. On OOD objects, GNR's SR increases from 12.81% to 44.33% as we increase the sample budget from 1 to 256.


Q3 — Can GNR use a GPU? Does MPC use a CPU?

GNR performance gains are agnostic to the selection of GPU and CPU. Both GNR and MPC use CPU for the retargeting comparison in Tab. 2 because SuperDex is a CPU-bound simulator to enable high-precision physics. GNR can also run with GPU-based simulation. We added clarifications in App. I where we compare GNR with SPIDER using GPU-accelerated MuJoCo Warp. GNR achieves 95% SR with 384 samples, compared with 96% for SPIDER with 16,384–32,768 samples.


Q4 — What is the difference between OOD objects and OOD geometries?

HOT3D has 33 distinct source objects (Fig. 11). OOD-object means the source object is unseen during training. OOD-geometry means the source object is seen during training and the test uses an unseen mesh variant (Fig. 12). We augment the original objects through methods including stretching and deformation (Fig. 12). The final dataset contains successful trajectories for 3.3k geometries across the 33 source objects (Tab. 16).


[1] Learning Dexterous Manipulation Using Contact Wrench Guidance From Human Demonstration

[2] SPIDER Scalable Physics-Informed DExterous Retargeting

[3] DexTrack: Towards Generalizable Neural Tracking Control for Dexterous Manipulation from Human References

The authors have revised and added experiments regarding the effectiveness of generated data on downstream learning methods, which experimentally verified that their neural retargeted data is beneficial towards robotic manipulation tasks. The concern regarding generalization from 23 source objects has also been explained using shared local geometries, augmentation, and test time scaling. Therefore, I believe this paper can be a valid reference for the dexterous manipulation community on scaling up data from humans, and have adjusted my overall evaluation accordingly.

AuthorsOpenReview ↗

Response to Reviewer xGTo

We are delighted to hear that our responses and revised draft have addressed your concerns, and we thank you for recommending acceptance!

Thank you for the thoughtful and constructive suggestions, and for acknowledging the importance of our research question of scaling up dexterous data and the value of GNR to the dexterous manipulation community.

To summarize, we added experiments demonstrating successful sim-to-real transfer with GNR-generated data on a highly challenging high-precision, long-horizon dexterous manipulation task, and showing the effectiveness of GNR for policy learning. Following your suggestion, we also provided insights into GNR's generalization: it results from both data scaling and test-time scaling, and objects with different overall geometries can share similar local geometry at contact points, which benefits from our real2sim pipeline that devlelops geometry data augmentation. We also added RL baseline experiments and GPU-accelerated results based on your comments.

Your feedback substantially strengthened both the completeness and the clarity of the paper. We truly appreciate your insight and time!

Reviewer_ww7SOpenReview ↗
Rating
3(+2)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper addresses the computational scaling bottleneck in physics-based human-to-robot retargeting for multi-fingered dexterous manipulation. While kinematic retargeting (IK) yields dynamically infeasible motions and sampling-based Model Predictive Control (such as DIAL-MPC and SPIDER) optimizes each demonstration in isolation from scratch, the authors hypothesize that dynamically feasible trajectories lie near a shared low-dimensional manifold. They propose Generative Neural Retargeting (GNR), a conditional Flow Matching model built on a temporal U-Net and Point Transformer V3 that predicts residual control corrections over IK references conditioned on reference motion and object state/geometry. GNR supports both open-loop test-time scaling (pass@S) and on-manifold warm-starting for closed-loop MPC (GNR+MPC). Integrated into a SuperDex Physics real-to-sim data engine, GNR scales retargeting across two axes: diversity (expanding HOT3D egocentric videos into 195k physically valid robot trajectories across 3.2k object geometries with dense 3D contact-force annotations) and difficulty (producing 3.4k trajectories across two millimeter-precision, long-horizon motion-capture tasks: ShapeFilter with 1.96 mm tolerance over 11 s, and NIST knob threading with <1 mm<1\text{ mm} tolerance over 29 s).

Critical strengths
  • Amortizing sampling-based retargeting via conditional flow matching and test-time scaling.
    Recasting isolated per-demo trajectory optimization into a residual Flow Matching generator directly addresses the scaling bottleneck of physics-based retargeting. The systematic trade-off analysis between seed dataset size NN and test-time sampling budget SS (Figure 5a & Table 9) is highly informative: on ShapeFilter, training on just N=50N=50 MPC seed trajectories and scaling to S=256S=256 boosts success from 13.0% to 99.5%, achieving an 11.2×11.2\times reduction in total dataset generation compute (765.6 vs. 8,541.0 CPU core-hours).
  • Pushing dexterous retargeting to millimeter-level precision and long horizons.
    Prior physics-based retargeting benchmarks (DexTrack, ManipTrans, SPIDER, CHORD) focus largely on short horizons or 20–150 mm tolerances (Table 3). Evaluating on ShapeFilter (1.96 mm, 11 s) and NIST assembly (1.0 mm, 29 s) exposes the boundary between where open-loop generation suffices and where closed-loop GNR+MPC refinement is indispensable.
Critical weaknesses
  • Lack of real-world hardware evaluation and ablation study on softened contact stiffness in simulation
    All results in the paper are evaluated exclusively in SuperDex Physics. In Appendix B.4 (lines 987–988), the authors disclose: "Contact stiffness affects the magnitude of the recorded forces. We use a value an order of magnitude below the simulator default, with lower finger joint gains." Reducing contact stiffness by 10×10\times together with compliant PD gains introduces substantial artificial compliance, which makes tight-tolerance insertion (ShapeFilter, 1.96 mm) and thread alignment/screwing (NIST, 1.0 mm) much more forgiving to contact collisions and jamming than on real hardware. Without either real-world execution on the Tesollo DG-5F hand or a sensitivity study across stiffer contact parameters, it remains uncertain how well the retargeted trajectories and contact-force labels transfer to physical robots.
  • Overclaimed scalability and asymmetric evaluation across both the diversity (HOT3D) and difficulty (NIST) axes, with unvalidated data-engine utility for policy learning.
    The paper's three central claims—outperforming MPC on diverse data (HOT3D), scaling to long-horizon high-precision tasks (NIST), and building a 195k-trajectory data engine for policy learning—are undermined by critical methodological flaws:
    1. Biased and leaky evaluation on HOT3D (Table 1 & App. E): The headline result (56.20% vs. 27.20%) splits the dataset by a single run of DIAL-MPC into MPC-success (27.2% of test set) and MPC-failure (72.8%) and assigns DIAL-MPC 0.00% SR on MPC-failure by definition without re-running it with a fresh random seed (ignoring regression-to-the-mean, where single runs of iCEM/CMA-ES solve 18.3%/16.8%). Moreover, GNR checks all S=256S=256 candidates against the binary success oracle Γ(ξ)=1\Gamma(\xi)=1 (pass@S), whereas sampling baselines evaluate 3,000 rollouts on a soft cost JJ and check Γ(ξ)=1\Gamma(\xi)=1 on only 1 final trajectory; under HOT3D's 2.15 s frozen-arm clips, 12-frame zero-gravity mid-air object weld (App. B.3), 10×10\times softened contact stiffness (App. B.4), and loose 150 mm/40∘150\text{ mm}/40^\circ tolerance, GNR actually yields worse object position/orientation tracking errors than iCEM and CMA-ES. Worse still, 45.9% of the test set (OOD-geometry) reuses the exact same 353 source human motions and 23 objects from GNR's training set; on truly unseen motions/objects (OOD-grasp/OOD-object), GNR's SR on MPC-failure drops from 45.1% to 25.8%–29.9% (Figure 11).
    2. Open-loop failure and hand-crafted MPC fallback on NIST (Section 5.3 & App. D): Because GNR predicts full-episode controls open-loop without state feedback, it collapses to 4.7% SR at S=256S=256 on the 29-second NIST assembly task (Figure 4b). The fallback GNR+MPC relies on a heavily hand-engineered 3-stage pipeline where Stages 1 and 2 (1,720 rollouts, 37.3 CPU core-h) are still executed from scratch by vanilla DIAL-MPC and GNR only warm-starts Stage 3, costing 73.9 CPU core-h per trajectory and yielding only a 1.8×1.8\times total compute reduction (Table 19).
    3. Unvalidated data-engine scaling and missing RL baselines (App. I): Neither the flagship 195k HOT3D dataset nor the dense contact-force labels are ever used to train a policy. The sole Behavioral Cloning test (App. I) only compares equal-sized 500-episode sets on ShapeFilter across 32 test episodes (14/32=43.75%14/32 = 43.75\% vs. 12/32=37.5%12/32 = 37.5\%), never testing whether expanding a small MPC seed set with GNR outperforms training on the MPC seeds alone, while omitting all learned RL retargeting baselines (DexTrack, ManipTrans).
Questions for authors
  • How sensitive are the retargeting success rates of DIAL-MPC and GNR (especially on ShapeFilter and NIST) when evaluated under the default (un-softened, 10×10\times stiffer) simulator contact stiffness or transferred to another physics engine (e.g., MuJoCo / Isaac Lab)? Have the authors tested replaying retargeted trajectories or BC policies on the physical Tesollo DG-5F hand?
  • On NIST, open-loop GNR gets 4.7% SR because open-loop errors compound over 29 seconds (580 control steps). Notice that on ShapeFilter, GNR is trained on 64-frame sequences with a 10-frame stride and conditioned on the arm state (Appendix G.2), whereas on NIST it is trained on complete 29-second episodes without state feedback. If GNR on NIST is instead trained on shorter windows conditioned on the current simulated robot/object state xtx_t and rolled out in a closed-loop receding-horizon manner (or with step-wise rejection sampling per stage), can GNR solve NIST without invoking DIAL-MPC?
  • Can the authors provide experiments showing that training a multi-object policy on the expanded 195k HOT3D dataset (95k MPC + 99k GNR) improves policy success rate on held-out geometries/grasps compared to training solely on the 95k MPC dataset? Additionally, how are the dense 3D contact-force annotations utilized in policy learning?
  • In Table 1, baselines like iCEM achieve lower median position/orientation tracking errors (Epos=26.04 mm,Erot=10.92∘E_{\text{pos}} = 26.04\text{ mm}, E_{\text{rot}} = 10.92^\circ on MPC-failure) than GNR (Epos=36.18 mm,Erot=17.58∘E_{\text{pos}} = 36.18\text{ mm}, E_{\text{rot}} = 17.58^\circ), yet GNR achieves a much higher success rate (42.58% vs. 18.27%). Could the authors provide a breakdown of the violation rates for each individual sub-criterion (γtrack,γpen,γgrasp,γforce\gamma_{\text{track}}, \gamma_{\text{pen}}, \gamma_{\text{grasp}}, \gamma_{\text{force}}) across methods to clearly show which physical constraints the sampling baselines violate?
8 comments
AuthorsOpenReview ↗

Response to Reviewer ww7S (Part I)

We thank the reviewer for the helpful comments, for acknowledging the importance of scaling up dexterous demonstrations, and for recognizing as a critical strength that our work pushes the boundary of high-precision (sub-millimeter), long-horizon retargeting on ShapeFilter and NIST. We have revised the paper and added experiments and clarifications to address the concerns. Real-robot demos are also available on our anonymized webpage.


W1: Real robot experiments

We added real-robot experiments. The policy trained on GNR data achieves zero-shot sim-to-real transfer. As the reviewer also noted, this pushes the boundary of dexterity. To our knowledge, this is the first retargeting study to achieve millimeter-precision sim-to-real policy transfer over long horizons, whereas existing works generally consider centimeter precision over a few seconds (e.g., CHORD, DexTrack, SPIDER). We added the sim-to-real results to Sec. 5.4. See video demos are on anonymized webpage.


W1 & Q1: Softened contact stiffness in simulation

To address the concerns about the simulation stiffness settings, we added new experiments and clarifications:

  1. Sim-to-real transfer. GNR achieves successful sim-to-real transfer (Sec. 5.4).

  2. Stiffness ablation (App. D.4.1). We ablate contact stiffness and find that GNR consistently outperforms IK and DIAL-MPC at all stiffness settings. Further details are in App. D.4. We lower the stiffness to reduce excessive simulated contact forces, which arise when fingers penetrate slightly into objects in simulators such as SuperDex and MuJoCo. For example, in a MuJoCo Warp replay of a SPIDER trajectory, the per-finger force exceeds 700 N while the hand lifts an object weighing 5 N. At the default SuperDex stiffness, IK replays exceed the 100 N per-finger force cap on about 47% of HOT3D demonstrations.

This is a common issue in modern simulators, yet contact-force magnitudes are often overlooked in retargeting studies. To produce more realistic force labels, we lower the default stiffness in SuperDex and set a per-finger force cap of 100 N. With this fixed cap, the SRs of IK, GNR, and DIAL-MPC all decrease as stiffness increases, because more trajectories exceed the force cap. Across the tested settings, GNR using 16 samples consistently outperforms DIAL-MPC using 3k samples by 12.9–15.5 percentage points.

  1. A different physics engine (App. I). We added results comparing SPIDER and GNR in a different physics engine, MuJoCo Warp, at its default stiffness. GNR consistently reduces the sampling budget while achieving high success.

W2.1: Evaluation

"Biased and leaky evaluation due to MPC-success and MPC-failure split. Missing multi-seed results"

Following the reviewer's suggestion, we revised Sec. 5.2 to clarify that the evaluation is neither biased nor leaky. In Tab. 1, we use 1k test episodes randomly sampled, and report unbiased aggregate results across three random seeds as suggested. The average SR of DIAL-MPC is 28.13% overall. On the MPC-failure subset defined by the initial run, the three new DIAL-MPC seeds achieve 10.3%, 9.6%, and 12.2% SR, averaging 10.7%. On the same subset, GNR achieves 42.6% SR with 256 candidates.

The MPC-success / MPC-failure split is only used as a generalization test to show that GNR, trained on a small subset of MPC-success trajectories, still generalizes to MPC-failure episodes (Fig. 3).

"GNR checks all candidates against the binary success oracle, whereas sampling baselines evaluate 3,000 rollouts on a soft cost and check on only 1 final trajectory"

To address the concern, we

  1. Added experiments comparing GNR using soft with MPC: we deploy GNR using the cost function to select a single trajectory from 256 smaples. On HOT3D, GNR using 256 samples achieves 34.2% SR when verifying only that rollout, compared with 28.1% SR for DIAL-MPC using up to 3,000 samples and also matches other baselines ~30% SR with 3,000 samples (Tab 1). In this comparison, both methods apply the success verifier only to their final output. GNR thus achieves a higher SR with a smaller proposal budget even when selecting by cost. We added these clarifications to App. B.6.

  2. We clarify that using a cost function is a structural requirement and standard practice for sampling-based optimizers. For example, optimizers such as DIAL-MPC and SPIDER generate the control sequence within each time window as a cost-weighted average over multiple samples, and then move on to the next time window with the averaged control sequence. The sequence cannot be verified until the final control sequence is generated through the cost-weighted averaging, and the cost is required to guide closed-loop sampling.

In contrast, GNR or IK generates full control sequences in open loop, so each candidate can be verified directly without closed-loop cost guidance.

AuthorsOpenReview ↗

Response to Reviewer ww7S (Part II)

... continued from Part I

W2.1: Evaluation

"Under HOT3D's 2.15 s frozen-arm clips, 12-frame zero-gravity mid-air object weld (App. B.3), softened contact stiffness (App. B.4), and loose tolerance, GNR actually yields worse object position/orientation tracking errors than iCEM and CMA-ES."

Q4: Reporting violation rates for each individual sub-criterion on HOT3D

We clarify that GNR and all baselines share the same clip length, welding, and stiffness settings for a fair comparison.

  1. As suggested by the reviewer, we report the violation rate of each sub-criterion in Tab. 1 across three random seeds, which is also shown below.
MethodStrack (%)pen (%)grasp (%)force (%)
IK156.329.437.115.9
DIAL-MPC3,00033.5 ± 1.134.5 ± 0.520.6 ± 1.025.2 ± 0.7
MPPI3,00035.2 ± 1.033.3 ± 1.021.6 ± 0.524.7 ± 0.4
iCEM3,00020.8 ± 0.237.9 ± 1.213.2 ± 0.630.9 ± 0.6
CMA-ES3,00025.7 ± 0.534.4 ± 0.919.5 ± 0.123.4 ± 1.0
GNR12824.5 ± 0.318.1 ± 0.212.1 ± 0.49.3 ± 0.6
GNR25622.4 ± 0.516.8 ± 0.310.9 ± 0.38.5 ± 0.2
  1. On HOT3D, we compare GNR's open-loop generation with the closed-loop optimizers iCEM and CMA-ES. With 256 samples, GNR has 4.3–8.7 mm higher median Epos and 4.0–4.8° higher median Erot than iCEM and CMA-ES with 3,000 samples, while achieving lower penetration, grasp, and force violation rates than both baselines. iCEM has the best tracking column and the highest penetration and force violation rates. As a result, GNR achieves 56.1% SR with 256 samples, compared with 34.9% for iCEM and 32.1% for CMA-ES with 3,000 samples. Improving success with fewer samples is our primary goal for scaling retargeting.

"Worse still, 45.9% of the test set (OOD-geometry) reuses the exact same 353 source human motions and 23 objects from GNR's training set; on truly unseen motions/objects (OOD-grasp/OOD-object), GNR's SR on MPC-failure drops from 45.1% to 25.8%–29.9% (Figure 11)."

  1. We clarify that this evaluation is not leaky. The geometries in OOD-geometry are unseen during training. This OOD-geometry split tests generalization to new object shapes, such as the stretched and deformed meshes shown in Fig. 12.

  2. MPC-failure episodes are expected to yield lower SRs. The 25.8%–29.9% cited by the reviewer is GNR’s SR on MPC-failure episodes, episodes that DIAL-MPC failed to solve with 3,000 samples. GNR learns only from MPC-success trajectories, yet it still generalizes to MPC-failure episodes that it never learns from. This is very helpful for expanding the dataset. For example, at S=256, GNR solves 127k of the 264k demonstrations in the MPC-failure set and significantly expands the dataset, even though the SRs are not as high as on MPC-success.


W2.2: Open-loop failure on NIST and hand-crafted MPC fallback

We first thank the reviewer for acknowledging, in the Critical Strengths, the difficulty of NIST (sub-millimeter precision over ~29 s), and for noting that existing retargeting methods typically use 20–150 mm precision requirements.

Inverse kinematics can be seen as an open-loop baseline, and it yields 0% SR on NIST, whereas open-loop GNR solves 4.7% of the NIST human trajectories. The low SR on this highly challenging task is expected and motivates the closed-loop GNR+MPC deployment.


"The fallback GNR+MPC relies on a heavily hand-engineered 3-stage pipeline where Stages 1 and 2 (1,720 rollouts, 37.3 CPU core-h) are still executed from scratch by vanilla DIAL-MPC and GNR only warm-starts Stage 3 … "

We clarify that GNR warm-starts the entire MPC pipeline starting from Stage 1, not only Stage 3, because Stage 3 depends on the outcome of Stage 2. Stages 1 and 2 are therefore not vanilla MPC. We revised App. C.3 to improve clarity.


"NIST yielding only a total compute reduction (Table 19)."

GNR initialization for MPC sampling significantly reduces both the per-trajectory optimization cost (from 795k to 266k CPU core-s, 3.0×) and the full dataset generation cost (1.8×).


Q2: Can we train GNR on NIST using 64-frame time windows and deploy GNR in a receding-horizon manner?

We thank the reviewer for the suggestion. We clarify that the main-paper open-loop results for both NIST and ShapeFilter already use 64-frame time windows, and we updated App. C.1 for clarity. Following the suggestion, we added experiments in App. B.5 that test closed-loop GNR deployment without MPC in a receding-horizon manner. The SRs are 4.7% for open-loop GNR and 5.7% for closed-loop GNR without MPC. This shows that sampling-based optimization is still necessary given the high-precision challenge of NIST, one of the hardest tasks in existing retargeting studies.

AuthorsOpenReview ↗

Response to Reviewer ww7S (Part III)

... continued from Part II

W2.3 & Q3: Data-engine scaling, policy learning, contact-force labels, and RL baselines

"Neither the flagship 195k HOT3D dataset nor the dense contact-force labels are ever used to train a policy. The sole Behavioral Cloning test (App. I) only compares equal-sized 500-episode sets on ShapeFilter across 32 test episodes"

We clarify that the main focus of this paper is making retargeting more scalable and efficient, because retargeting is itself a hard problem for dexterous manipulation and suffers from a scalability bottleneck. BC policy learning and the use of force labels for BC are neither our claim nor our focus. Many state-of-the-art retargeting methods likewise focus primarily on retargeting itself (e.g., SPIDER, DexMachina, CHORD, Do As I Do). BC policy learning for dexterous manipulation is another hard problem, and whether force labels help and how to use them remain very open problems in the community.

However, we thank the reviewer for raising the question. We are happy to extend the BC experiments based on the reviewer's suggestion (Sec. 5.4, App. J.1):

  1. We trained BC policies on the GNR, GNR+MPC, and MPC datasets, each with 3 random seeds, and evaluated every 50k steps on 500 episodes per evaluation. GNR data alone achieves similar success to MPC data while reducing the data generation cost by 11.2×, and GNR+MPC data improves SR by around 7% over MPC data at 4× lower generation cost.
  2. More importantly, a visuomotor policy trained in simulation without any real-robot demonstrations, succeeds in sim-to-real transfer on a millimeter-precision, long-horizon task (Sec 5.4), which the reviewer also noted pushes the boundary of dexterous manipulation.

"never testing whether expanding a small MPC seed set with GNR outperforms training on the MPC seeds alone"

Q3: "BC learning a multi-object policy on the expanded HOT3D dataset (MPC+GNR) improves policy compared to BC on MPC data alone. Additionally, how are the dense 3D contact-force annotations utilized in policy learning?"

Although BC or using force for BC is not our primary focus, we're happy to add HOT3D BC, and contact force BC experiments based on the suggestion (App. J.1.2).

  1. GNR Improves BC on HOT3D by Expanding Data Coverage. We randomly sample 5k MPC demonstrations as the seed training set, and 3.7k held-out test epsidoes. We then expand the training data as suggested with 5k MPC or 5k GNR demonstrations, and then with 15k MPC or 15k GNR demonstrations, to compare policy SRs. BC policies trained on the MPC seed + GNR data outperform policies trained on the MPC data with the same amount of demonstrations on both MPC-success and MPC-failure test episodes:
Training dataTotal demosMPC-failure SR (%)MPC-success SR (%)
5k MPC (seed)5k33.850.5
+ 5k MPC10k33.151.1
+ 5k GNR10k36.153.5
+ 15k MPC20k32.052.8
+ 15k GNR20k36.853.7
  1. Studied BC with Contact Forces. The use of contact force in policy learning is a very open question (whether it helps, and how to use it [1]). We ran experiments on ShapeFilter with visuomotor diffusion policies trained on 2k demonstrations with three seeds each (App. J.1.1). Using force as an auxiliary prediction objective raises SR from 27.4 ± 1.9% to 31.1 ± 2.0%, and using force as an additional observation performs similarly to the policy without force (26.2 ± 2.2%).

"… omitting all learned RL retargeting baseline …"

We thank the reviewer for the suggestion. We have added RL baseline results to App. B.2 (Tab. 5). RL is typically more costly and requires more samples for policy optimization. We use DexTrack and ManipTrans as baselines as suggested. We cap the environment steps during training. At a matched budget on HOT3D, GNR achieves 56.1% SR with 256 samples (13k environment steps), compared with 53.3% for DexTrack and 51.7% for ManipTrans with 162k steps. GNR also reduces compute cost by 32x and 73x compared to DexTrack and ManipTrans, respectively. Here, we use the same seed dataset to initialize GNR, and the shared model in DexTrack, and ManipTrans to ablate the learned prior.


[1] FACTR: Force-Attending Curriculum Training for Contact-Rich Policy Learning. arXiv:2502.17432, 2025.

[2] SPIDER Scalable Physics-Informed Dexterous Retargeting

[3] Learning Dexterous Manipulation Using Contact Wrench Guidance From Human Demonstration

Reviewer_be2pOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents GNR, a flow matching model trained on MPC-generated, simulation-validated trajectories to accelerate physics-aware human-to-robot motion retargeting. The method learns a prior over successful trajectories and uses it for direct generation or MPC initialization.

Critical strengths
  • The idea of learning a neural prior to accelerate physics-based optimization is interesting and well motivated.
  • The experiments cover diverse objects, robot embodiments, and high-precision manipulation tasks. The reported improvements in computational efficiency are promising.
  • The paper is clearly written and well organized.
Critical weaknesses
  • The main limitation is the lack of real-world evaluation. Since physical feasibility is a central motivation, even replaying the retargeted trajectories on real robots would significantly strengthen the paper. Without real-world validation, it is difficult to judge whether the simulation physics accurately reflects real-world interactions.
  • The experiments do not fully isolate the contribution of the learned cross-episode prior. While GNR improves optimization efficiency, it remains unclear how much of the improvement comes from learning shared knowledge across episodes versus accelerating optimization within individual episodes.
  • The HOT3D evaluation uses a customized success criterion, with relatively loose object-tracking thresholds of 150 mm and 40 degrees. Although additional contact and penetration constraints are included, the use of custom metrics makes direct comparison with prior work difficult. Reporting established metrics, such as ADD-AUC in SPIDER, would strengthen the evaluation.
Questions for authors
  • Can the authors further investigate how much neural learning accelerates optimization within a single episode, without using a cross-episode prior? This would help clarify the contribution of learning shared information across demonstrations.
  • Can the authors provide real-world replay experiments, even on a limited number of trajectories? This would help validate the physical feasibility of the generated motions.
  • Can the authors report established evaluation metrics, such as ADD-AUC, on HOT3D? It would also be helpful to clarify how the customized success criterion relates to those used in previous works, particularly SPIDER, to make a more direct comparison.
4 comments
AuthorsOpenReview ↗

Response to Reviewer be2p

We thank the reviewer be2p for the thoughtful and constructive comments! We are pleased that the reviewer found our idea interesting and well motivated and our experimental results promising.

We have revised the paper according to the reviewer's suggestions, which have helped us improve the clarity of the paper and strengthen the experiments. New real-robot demos are also included on the anonymous webpage for review. https://gnr-review.github.io/


W1 / Q2 — Real-world evaluation

We conducted real-robot experiments based on your suggestion. We are very pleased to show that a policy trained using GNR data can successfully achieve zero-shot sim-to-real transfer. We added the real-robot evaluation to the main paper as a major new result (Section 5.4). To our knowledge, this is also the first result in retargeting studies that achieves millimeter-precision policy sim-to-real transfer over long horizons for five-fingered dexterous manipulation. Existing methods primarily operate with 20–150 mm precision over a few seconds (e.g., CHORD and SPIDER).


W2 / Q1 — Isolating the improvements via cross-episode learning vs. within-episode acceleration

We added new experiments and discussion in Appendix B.1 to show the benefit of cross-episode learning. Specifically, (1) Fig 5 demonstrates that through cross-episode learning using more data, GNR performance improves on the test set monotonically, and (2) we added new experiments in Tab 4 training GNR within each episode. The cross-episode prior improves success from 78.1% to 97.4% when using 64 samples.


W3 / Q3 — HOT3D success criterion and ADD-AUC10

  1. We added clarification about the choice of 150 mm and 40°, which follows the practice in CHORD for egocentric videos, where accurate references are hard to obtain and tracking thresholds are therefore higher.
  2. We added new experiments to Tab. 7 showing that our conclusions for GNR hold under tighter criteria. At 100 mm and 30°, GNR achieves 29.9% vs. 23.7% for DIAL-MPC. At 50 mm and 20°, GNR achieves 18.9% vs. 14.5%. This is around 30% higher relative success, even though GNR was trained on data generated under the original 150 mm requirement.
  3. We added ADD-AUC10 and detailed criteria to Table 1, as suggested by the reviewer. We clarify that GNR in Table 1 is open-loop, while sampling-based baselines directly optimize tracking through iterative sampling and refinement. GNR has the lowest finger-tracking error among all methods, and the lowest penetration, grasp, and force violation rates. Its ADD-AUC10 is close to that of its seed generator DIAL-MPC (58.53 vs. 58.88), with about 7.6× lower CPU cost.

[1] Learning dexterous manipulation using contact wrench guidance from human demonstration

AuthorsOpenReview ↗

Rebuttal Summary for Reviewer be2p

We thank Reviewer be2p for the thoughtful and constructive feedback, which has helped us improve the clarity and experiments of the paper. We also thank the reviewer for finding our method interesting and well motivated, the experiments diverse and improvements promising.

As the discussion period comes to a close, we would love to hear any further feedback, and we are happy to address any additional questions.


We summarize our revision below:

  1. Added real-world evaluation (W1, Q2). We added real robot experiments based on the reviewer's suggestion. A vision-based policy learned using GNR data achieved successful sim2real transfer without any real-robot demonstrations to a five-fingered dexterous hand on a millimeter-precision, long-horizon task (see Sec. 5.4 and webpage videos).
  2. Added ablation for cross-episode prior (W2, Q1). We added experiments and discussions in App. B.1 to show that cross-episode prior is the key to GNR's success, raising SR from 37.8% to 92.3% at 16 GNR samples (App. B.1). Our data scaling results in Fig. 5 further confirm it: e.g., SR at one GNR sample rises from 11.5% to 52.4% as training demonstrations grow from 50 to 2.5k providing stronger cross-episode prior.
  3. Added ablation of HOT3D success criterion and reported more comprehensive evaluation metrics (W3, Q3). Based on the suggestion, we revised Tab 1 to provide several evaluation metrics as suggested. We show that open-loop GNR yields the lowest penetration, grasp, and force violation rates, and achieves comparable ADD-AUC and tracking to its seed generator with 7.6x lower compute cost. We added ablation experiments for the success criterion (using 150mm/40°, 100 mm/30°, 50 mm/20°) and show that GNR's gains are consistent under tighter thresholds compared to baselines.

Thanks authors for the reply and additional experiments. A few of my original concerns remain:

  1. The purpose of the “real-world evaluation” presented in the paper is not yet clear to me. It uses a vision-based diffusion policy with RGB cameras, while the paper focuses on a neural prior to speed up physics-aware retargeting. I am unsure how GNR improves RGB-based sim-to-real transfer and its precision. My original point was to check whether the physics parameters reasonably match real-world through diverse trajectory replays, as in SPIDER.

  2. My original question was whether GNR can outperform MPC within one new episode. If so, this would show that GNR can generally speed up physics-based retargeting for any new episodes, even without a prior. The authors instead show that a prior outperforms no prior, so perhaps the paper’s focus is the large-scale prior.

  3. My original point was to compare with numbers reported in previous papers to avoid customized benchmark settings. The authors added metrics, but the numbers still do not appear directly comparable to previous papers.

AuthorsOpenReview ↗

Response to Reviewer be2p

Nov 18 update: As the discussion period comes to a close, we would like to thank the reviewer for their time and valuable suggestions, which have greatly helped us improve the clarity of our paper!


We thank the reviewer for the follow-up questions. We clarify the three points below.

1. Real-world evaluation and physics fidelity

We thank the reviewer for the thoughtful question. We chose policy evaluation to address the concern about physics fidelity because a closed-loop policy validates it more reliably, and we revised the paper with this discussion in App B.7.

Open-loop replay is sensitive to small errors. A failed real-world replay could therefore reflect reset errors rather than physics fidelity, espeically on challenging tasks such as ShapeFilter, which require millimeter-level precision over a long horizon. Small errors in the initial object pose and robot actuation accumulate without feedback during the replay.

A closed-loop policy can better isolate whether the simulated contact dynamics transfer. Our visuomotor policy is trained only on GNR+MPC trajectories generated in simulation, with no real-robot demonstrations. The RGB images are rendered observations and do not alter the trajectories or physics, so the policy learns its contact behavior entirely from simulated physics. Its real-robot SR (30%) is close to its simulated SR (38%) despite the challenging visual gap, which indicates that the simulated contact dynamics match the real system (Sec. 5.4).

Prior retargeting works use both trajectory replay and policy learning to verify physics fidelity [3], while policy evaluation is generally harder because of the sim2real gap and the requirements for retargeting larger-scale data.

As the reviewer mentioned that the paper focuses on speeding up physics-aware retargeting. On ShapeFilter, we achieved 4x-11.2x speed up compared to MPC-based generation when producing the policy learning data, aligning with our focus.

We hope these clarifications address the reviewer's concern about physics fidelity.


2. GNR within a single new episode

We agree with the reviewer's reading: GNR's focus is the prior learned across demonstrations. GNR is designed to amortize physics-based retargeting, and its speedup on each new episode comes from this prior (cross-episode ablation in App. B.1 and data scaling in Fig. 5). Importantly, the prior does not need to be large. On ShapeFilter, GNR trained on only 50 MPC demonstrations reaches 99.5% SR on unseen episodes at S=256 through test-time scaling (Fig. 5).

As suggested by the reviewer, we clarify that we tested MPC and GNR on new episodes containing unseen objects, geometries, and motions (Fig. 11–13). GNR speeds up retargeting for each new episode using the learned prior, and outperforms or matches MPC with far fewer samples across all three tasks:

  • HOT3D. All test episodes contain an unseen object, grasp, or geometry. GNR achieves 56.1% compared with 28.1% for DIAL-MPC while lowering CPU cost by 7.6× lower (Tab. 1). GNR also solves 31.9–58.4% of the new episodes on which MPC fails (Fig. 3).
  • ShapeFilter and NIST (unseen poses). GNR speeds up dataset generation on new episodes and reduces total cost by 11.2× and 1.8x.

3. Comparison with numbers reported in prior work

We thank the reviewer for the suggestion and provide further clarification.

To compare directly with prior work in its own setting as suggested by the reviewer, we evaluate GNR on SPIDER's multi-embodiment data. We revised the paper and added App. I.2. GNR consistently reduce sampling cost while achieve comparable SRs and ADD-AUC10 across 5 different hands.

Our benchmarks. HOT3D, ShapeFilter, and NIST are benchmarks introduced in our work, so we reproduce prior optimizers on them. As the reviewer suggested, we added the established ADD-AUC10 metric for all methods in Tab. 1.

HandSR (%) GNRSR (%) SPIDERADD-AUC10 GNRADD-AUC10 SPIDER
Inspire1009567.558.0
Schunk1009561.963.9
XHand10010063.266.4
Ability759555.962.1
Allegro1009558.662.7
Mean959661.462.6

[1] More than a Million Ways to Be Pushed: A High-Fidelity Experimental Dataset of Planar Pushing. IROS, 2016.

[2] Predictable Behavior During Contact Simulation: A Comparison of Selected Physics Engines. Computer Animation and Virtual Worlds, 2016.

[3] Morphometric Imitation From Morphology and Contact Aware Hand Retargeting to Sim-to-Real Visuomotor Policy

Comments on the paper

2 comments

As we near the end of the discussion period for this submission, please check whether the authors have answered questions or responded to critiques. If they have, we encourage the reviewers to leave a comment for the authors. If reviewers have raised new points, a gentle nudge to the authors to respond would also be appreciated.

AuthorsOpenReview ↗

Rebuttal Summary

To the Area Chair and Reviewers:


We thank the reviewers for their thoughtful feedback and constructive engagement. We have revised the paper (changes in blue) and updated the anonymized webpage (https://gnr-review.github.io/) with new real-robot demos based on the suggestions.

GNR aims to address the scalability bottleneck of human-to-robot retargeting to generate large-scale dynamically feasible dexterous robot trajectories. We solve two tasks that are among the hardest dexterous retargeting tasks, with millimeter precision over long horizons, and create a diverse dataset of 223k robot trajectories, one of the largest human-to-robot dexteorus datasets. GNR enables zero-shot sim2real transfer of a learned visuomotor policy to a five-fingered robot hand.


We thank the reviewers for recognizing the importance of making retargeting scalable (ww7S, xGTo), the well-motivated and interesting idea (be2p, xGTo), and that our work pushes the boundary of dexterous retargeting to sub-millimeter precision over long horizons, whereas existing works typically consider 20–150 mm over a few seconds (ww7S).

We are grateful for the many helpful suggestions, which have improved the rigor and clarity of our paper. After the discussion, all three reviewers rated the paper 3. Reviewers xGTo and ww7S indicated that their concerns have been addressed and raised their scores to 3.


We summarize several key points in responses and revision as below:

  1. Successful zero-shot sim-to-real on a millimeter-precision, long-horizon task. (be2p, ww7S, xGTo)

We added real robot experiments as suggested. A visuomotor policy trained on GNR generated simulation data, with zero real robot demonstrations, achieves successful sim2real transfer to a five-fingered dexterous hand on a challenging task (1.96 mm tolerance, 11 s). See Sec. 5.4, and the webpage videos for demos.

  1. Demonstrating GNR's effectiveness for policy learning. (ww7S, xGTo)

We added more BC experiments. Policy learning using GNR data matches MPC data in success rates (SRs) while reducing generation cost by 11.2x. GNR+MPC data improves SR by 6.1 pp while reducing generation cost by 4x (see Sec. 5.4 and F.6). We added multi-seed results of BC using 500 evaluation episodes per checkpoint based on reviewers' suggestions. On HOT3D, expanding the dataset with GNR data outperforms adding the same amount of MPC data by 3-4.8 pp (App. J.1.2). We also test BC with force labels (suggested by ww7S). Using force prediction as auxiliary objectives raises SR by 3.7 pp (App. J.1.1). This addresses the concerns regarding the effectiveness of GNR for downstream policy learning and force labels, although our primary focus remains scalability and efficiency of human-to-robot retargeting itself.

  1. GNR is more sample-efficient than RL. (ww7S, xGTo): We added two RL baselines. On HOT3D, GNR reaches 56.1% SR with 13k environment steps, compared with 53.3% (DexTrack) and 51.7% (ManipTrans) using 162k steps per demonstration, at 32× and 73× higher CPU cost than GNR.

  2. HOT3D results with more metrics. (be2p, ww7S): We added 3-seed results with more detaild metrics (ADD-AUC10, force/penetration violation) to Tab. 1 . GNR reaches 56.1% SR versus 27.5–34.9% for four sampling-based baselines, using 8.5% of their samples (Sec. 5.2).

  3. Gains are not limited to a CPU-bound simulator. (ww7S): We added clarifications GNR is applicable to GPU-accerated simulator (e.g., MuJoCo Warp). SuperDex is used to produce demonstrations with high-precision physics. In MuJoCo Warp GNR reaches 95% SR on SPIDER's benchmark with 384 samples, compared with 96% for SPIDER with 16k–32k samples (App. I).

  4. Ablation of different contact stiffness settings. (ww7S): We achieved sim2real transfer with our simulation settings. We also added ablation experiments of stiffness settings to show that GNR outperforms DIAL-MPC by 12.9–15.5 pp (App. D.4.1) under all different stiffness settings. Our stiffness settings also produce more realistic force labels (App. D.4.1).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.