- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
DexForge converts human manipulation demonstrations into executable robot trajectories through visual reconstruction, contact-aware kinematic retargeting, and differentiable dynamics optimization. It jointly adjusts contact anchors and robot configurations to accommodate embodiment differences, then uses these contacts to guide trajectory refinement. The framework also jointly optimizes spherical Gaussian appearance and geometry and provides an accelerated differentiable collision interface integrated with existing dynamics. Evaluation covers 130 demonstrations from DexYCB and HOT3D across seven robot hands, with component ablations, SPIDER comparisons, sim-to-sim transfer, and real-world grasp-and-hold demonstrations.
Critical strengths
-
Useful contact adaptation across embodiments. Jointly refining contact anchors and robot configurations addresses a concrete limitation of directly transferring human contacts. The improvements obtained by adding this stage to SPIDER provide evidence that the contact guidance is useful beyond the complete DexForge pipeline.
-
Broad simulation evaluation. Experiments span two datasets and seven hands, including a stronger baseline that shares the proposed kinematic references and contact guidance. This provides meaningful evidence of improved retargeting performance across embodiments.
Critical weaknesses
-
The central methodological advance is insufficiently established. The paper presents several individually motivated components, but the contribution currently reads primarily as an integration of contact adaptation, Gaussian modeling, and existing differentiable dynamics and optimization techniques. The incremental advance over the related contact-refinement and physics-based retargeting methods needs clearer substantiation. In particular, the Gaussian representation is presented as a major contribution, yet its downstream benefit over alternative geometry representations is not isolated under the same retargeting pipeline. The experiments establish the effectiveness of the assembled system more clearly than they establish the necessity and significance of its central design choices.
-
Validation of high-fidelity reproduction remains limited in important settings. Real-world evaluation measures stable grasp-and-hold rather than reproduction of demonstrated object trajectories. Moreover, although HOT3D includes in-hand manipulation, the evaluation does not separately characterize performance under frequent contact changes or sliding contacts, which are particularly relevant to the stable-anchor formulation. These gaps limit confidence in how broadly the demonstrated precision extends beyond the evaluated simulation settings.
Questions for authors
Could you clarify which methodological advance you consider central relative to prior contact-refinement and differentiable retargeting approaches? If available, a comparison using alternative collision representations within the same pipeline would help establish the specific value of the Gaussian formulation.
Could you share any available analysis of sequences with frequent contact changes or sliding contacts, and any object-trajectory measurements from real-world execution? Such evidence would help assess the scope of the high-fidelity reproduction claim.
7 comments
Response to Reviewer 4CZr [2/3]
Gaussian formulation. The Gaussian formulation is a supporting contribution: we chose Gaussians because one representation of both appearance and geometry can be built directly and conveniently from video, avoiding two separate representations. This unified representation provides both the appearance needed for tracking and the collision geometry for simulation, with differentiable collision detection. Its specific value is that it improves reconstruction, tracking, and retargeting accuracy over a mesh built from the same frames (table below); it is, however, not irreplaceable in our retargeting pipeline, which supports any representation that provides differentiable collision detection. We have revised the contribution statement accordingly (Sec. 1 of the revised PDF): robot-adapted stable contacts as the central advance, and the Gaussian formulation as a supporting contribution.
To show the compatibility of our method and the specific value of the Gaussian formulation, on the DexYCB ablation subset with the Sharpa and LEAP hands (5 sequences × 2 hands, five seeds), from the same real RGB-D frames, we reconstruct both Gaussians and a mesh (made watertight and colored), track each with a suitable tracker (FoundationPose with the same eight views for the mesh), and run our retargeting pipeline. We also add a group in which the mesh uses the same trajectories as the Gaussians, to isolate the effect of the collision representation on the retargeting pipeline.
| Route | near-surface SDF error vs dataset mesh | PSNR | tracking (center / rotation) | retargeting vs tracked reference (E_pos / E_rot on successes / success rate) | retargeting vs GT (E_pos / E_rot on successes / success rate) |
|---|---|---|---|---|---|
| Our Gaussians | 2.77 mm | 20.21 dB | 3.78 mm / 3.14° | 1.20 mm / 0.30° / 80% | 2.05 mm / 1.63° / 60% |
| Our tracking + reconstructed-mesh collision | 3.75 mm | 20.21 dB | 3.78 mm / 3.14° | 1.52 mm / 0.31° / 80% | 2.31 mm / 1.63° / 60% |
| Reconstructed mesh + FoundationPose + mesh collision | 3.75 mm | 16.80 dB | 6.20 mm / 5.95° | 1.73 mm / 0.41° / 80% | 2.90 mm / 3.08° / 60% |
Comparing rows 1 and 2, switching only the collision representation from Gaussians to mesh causes only a small decrease in retargeting. Comparing rows 1 and 3, our Gaussians provide a better visual representation during reconstruction (PSNR 20.21 vs 16.80 dB), with better tracking quality (3.78 mm / 3.14° vs 6.20 mm / 5.95°).
On the same real RGB-D data, we also compare with other Gaussian constructions. Our formulation gives the most accurate collision geometry, and its retargeting performance is better than ContactGaussian-WM and far better than spherical 3DGS:
| Gaussian construction | near-surface SDF error vs dataset mesh | tracking (center / rotation) | retargeting vs tracked reference (E_pos / E_rot on successes / success rate) | retargeting vs GT (E_pos / E_rot on successes / success rate) |
|---|---|---|---|---|
| Ours | 2.77 mm | 3.78 mm / 3.14° | 1.20 mm / 0.30° / 80% | 2.05 mm / 1.63° / 60% |
| ContactGaussian-WM | 3.26 mm | 3.70 mm / 3.28° | 1.51 mm / 0.42° / 80% | 3.22 mm / 2.47° / 60% |
| Spherical 3DGS | 9.43 mm | 7.90 mm / 7.41° | 2.93 mm / 0.69° / 40% | 2.20 mm / 2.08° / 20% |
Response to Reviewer 4CZr [3/3]
Q2: Could you share any available analysis of sequences with frequent contact changes or sliding contacts, and any object-trajectory measurements from real-world execution? Such evidence would help assess the scope of the high-fidelity reproduction claim.
A2. Thank you for this suggestion.
Frequent contact changes and sliding. We binned the HOT3D sequences into tertiles by contact-switch rate (contact starts and ends summed over five fingers) and by the fraction of sliding frames (tangential speed above 20 mm/s).
| Success (%): DexForge / Contact-aware+SPIDER | low | medium | high |
|---|---|---|---|
| contact-switch rate (none / 0.2–1.4 / ≥ 1.5 per second) | 44.2 / 1.3 | 37.6 / 0.4 | 20.9 / 0.0 |
| sliding-frame ratio (≤ 45% / 45–56% / > 56%) | 51.0 / 1.0 | 20.4 / 0.2 | 33.2 / 0.6 |
Success decreases as contact switching becomes more frequent, but the stable-anchor formulation still handles these sequences: DexForge stays far above the baseline in every tertile, reaching 20.9% versus 0.0% in the tertile with the most frequent contact changes. Sliding shows no monotonic relation; slow and moderate sliding is segmented into consecutive anchor intervals.
To evaluate the stable contact anchors on the hardest sequences, we took the 10 HOT3D sequences with the most frequent contact changes (3.0 to 9.0 contact switches per second; LEAP and Sharpa hands, five seeds) and compared the full method with a variant that removes contact guidance from both the kinematic and dynamics stages:
| 10 most frequent contact-switch sequences | 10 mm / 0.1 rad (E_pos / E_rot / success rate) | 20 mm / 0.2 rad (E_pos / E_rot / success rate) |
|---|---|---|
| Full method | 5.32 mm / 1.92° / 20% | 8.70 mm / 2.56° / 40% |
| Full method (without contact guidance) | — / — / 0% | 15.58 mm / 4.13° / 25% |
These results show that, even on sequences with frequent contact switching, our stable anchors still bring a significant improvement rather than limiting performance, since each finger can have different anchors in different contact intervals and can thus represent frequent contact switching.
Real-world object trajectories. Because motion-capture markers on the object would affect the grasp, we measured the object trajectories during real-robot execution with an additional calibrated RealSense D456 camera and FoundationPose++, and added Contact-aware+SPIDER as a baseline under the same open-loop replay (five objects, 10 trials each). Errors are against the dataset ground truth and defined on successful trials.
| Criterion | DexForge (E_pos / E_rot / success rate) | Contact-aware+SPIDER (E_pos / E_rot / success rate) |
|---|---|---|
| grasp-and-hold (paper) | — / — / 80% | — / — / 64% |
| 20 mm / 0.2 rad | 13.47 mm / 7.45° / 38% | 17.14 mm / 8.78° / 14% |
| 40 mm / 0.4 rad | 23.31 mm / 12.14° / 74% | 27.62 mm / 16.78° / 44% |
DexForge outperforms the baseline at every criterion. These results are now in Sec. 4.3 (Table R1) and Appendix J of the revised PDF.
We hope these results address your concerns; if so, we would be grateful if you would consider updating your assessment.
Response to Reviewer 4CZr [1/3]
Thank you for your valuable comments. We respond to your questions in turn and hope our responses address your concerns; if any remain, we welcome further discussion. In the revised PDF, changes responding to your comments are marked in blue (changes shared with other reviewers in teal): Sec. 1-2, Sec. 4.2.2, Sec. 4.3 (Table R1), and Appendices G, J, K (Tables R2-R3) and M (Tables R6-R7).
Q1: Could you clarify which methodological advance you consider central relative to prior contact-refinement and differentiable retargeting approaches? If available, a comparison using alternative collision representations within the same pipeline would help establish the specific value of the Gaussian formulation.
A1. Thank you for this question.
Central advance. Our central advance is to jointly optimize contact anchors and robot key configurations during kinematic retargeting to obtain robot-adapted stable contacts; these contacts greatly improve the quality of the kinematic retargeting results, enabling gradient-based local optimization, which depends on its reference trajectory, to achieve high-fidelity retargeting.
Representative prior contact-refinement methods do not optimize contact anchors and robot key configurations jointly: C2Dex, SPIDER, and Lakshmipathy et al. directly use contacts extracted from the human hand without optimizing the contact anchors, while GenHand targets single-frame grasps and first optimizes the anchors alone before solving for the robot configuration. These methods may force the robot to fit unreachable anchors, producing unreasonable contacts. A new sequential ablation confirms this: refining the anchors first and then solving the robot key configurations with the anchors fixed reaches only 44.6% on DexYCB and 10.3% on HOT3D, below even the raw human anchors (63.4% / 18.9%) and far below joint optimization (78.9% / 32.0%). In our observation, this is because optimizing the anchors alone, unconstrained by the hand's configuration, often moves them far outside the hand's reachable region. In contrast, while optimizing the contact anchors we account for the differences between the human hand and the robot configuration, obtaining robot-adapted stable contact anchors that follow the human contact intent. These anchors are optimized for reachability, collision avoidance, and force closure to obtain high-quality kinematic retargeting results. The table below isolates each design choice.
Success rates averaged over seven hands (Table 1); the sequential variant is new (Sec. 4.2.2 and Appendix G of the revised PDF).
For differentiable retargeting, gradient-based methods are efficient and precise, but contact gradients are unreliable in complex contact and the optimization is local, so they cannot search over contact modes as sampling methods can. Prior methods stabilize the optimization by softening gradients or through annealing curricula (QuasiSim), but do not optimize the input reference with respect to contact. We instead optimize stable contact anchors already in kinematic retargeting, which largely determines the contact mode, so that gradient-based optimization only refines the motion under a good reference, with the anchors remaining its interaction targets, instead of exploring contact modes or making large updates, which gradients do poorly. This is why our gradient-based optimization achieves high precision. This core advance also improves other physics-based retargeting methods (e.g., Contact-aware+SPIDER in the paper), so it is not limited to gradient-based retargeting.