acceptodds
Under review as a conference paper at ICLR 2027

DexForge: High-Fidelity Physics-Informed Dexterous Retargeting

Abstract

Human demonstrations offer rich examples of precise dexterous manipulation and a promising source of robot training data. However, high-fidelity reproduction of demonstrated motions and hand–object interactions across robot embodiments remains challenging under physical constraints. We present DexForge, a differentiable physics-grounded framework for converting human video demonstrations into high-fidelity robot trajectories. We reconstruct spherical-Gaussian object models and hand–object motion from visual observations, then build a differentiable simulator combining efficient Gaussian collision detection with existing differentiable dynamics. Based on this simulator, DexForge combines contact-aware kinematic retargeting with force-aware dynamics retargeting: robot-adapted stable contacts guide kinematic reference construction and subsequent gradient-based control refinement for precise physical motion reproduction. Experiments on 130 DexYCB and HOT3D demonstrations across seven dexterous hands show success-rate gains of approximately 35–53 percentage points over the baseline, with object position and orientation tracking errors on successful trajectories reduced by approximately 34–67% and 71–78%, respectively. Further experiments demonstrate open-loop transfer to MuJoCo and real-robot execution. Our project page is available at https://dexforge-42306.github.io/project-page/.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Dexterous manipulation, human-to-robot retargeting, differentiable simulation, contact adaptation, trajectory optimization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_4CZrOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

DexForge converts human manipulation demonstrations into executable robot trajectories through visual reconstruction, contact-aware kinematic retargeting, and differentiable dynamics optimization. It jointly adjusts contact anchors and robot configurations to accommodate embodiment differences, then uses these contacts to guide trajectory refinement. The framework also jointly optimizes spherical Gaussian appearance and geometry and provides an accelerated differentiable collision interface integrated with existing dynamics. Evaluation covers 130 demonstrations from DexYCB and HOT3D across seven robot hands, with component ablations, SPIDER comparisons, sim-to-sim transfer, and real-world grasp-and-hold demonstrations.

Critical strengths
  1. Useful contact adaptation across embodiments. Jointly refining contact anchors and robot configurations addresses a concrete limitation of directly transferring human contacts. The improvements obtained by adding this stage to SPIDER provide evidence that the contact guidance is useful beyond the complete DexForge pipeline.

  2. Broad simulation evaluation. Experiments span two datasets and seven hands, including a stronger baseline that shares the proposed kinematic references and contact guidance. This provides meaningful evidence of improved retargeting performance across embodiments.

Critical weaknesses
  1. The central methodological advance is insufficiently established. The paper presents several individually motivated components, but the contribution currently reads primarily as an integration of contact adaptation, Gaussian modeling, and existing differentiable dynamics and optimization techniques. The incremental advance over the related contact-refinement and physics-based retargeting methods needs clearer substantiation. In particular, the Gaussian representation is presented as a major contribution, yet its downstream benefit over alternative geometry representations is not isolated under the same retargeting pipeline. The experiments establish the effectiveness of the assembled system more clearly than they establish the necessity and significance of its central design choices.

  2. Validation of high-fidelity reproduction remains limited in important settings. Real-world evaluation measures stable grasp-and-hold rather than reproduction of demonstrated object trajectories. Moreover, although HOT3D includes in-hand manipulation, the evaluation does not separately characterize performance under frequent contact changes or sliding contacts, which are particularly relevant to the stable-anchor formulation. These gaps limit confidence in how broadly the demonstrated precision extends beyond the evaluated simulation settings.

Questions for authors

Could you clarify which methodological advance you consider central relative to prior contact-refinement and differentiable retargeting approaches? If available, a comparison using alternative collision representations within the same pipeline would help establish the specific value of the Gaussian formulation.

Could you share any available analysis of sequences with frequent contact changes or sliding contacts, and any object-trajectory measurements from real-world execution? Such evidence would help assess the scope of the high-fidelity reproduction claim.

7 comments
AuthorsOpenReview ↗

Response to Reviewer 4CZr [1/3]

Thank you for your valuable comments. We respond to your questions in turn and hope our responses address your concerns; if any remain, we welcome further discussion. In the revised PDF, changes responding to your comments are marked in blue (changes shared with other reviewers in teal): Sec. 1-2, Sec. 4.2.2, Sec. 4.3 (Table R1), and Appendices G, J, K (Tables R2-R3) and M (Tables R6-R7).

Q1: Could you clarify which methodological advance you consider central relative to prior contact-refinement and differentiable retargeting approaches? If available, a comparison using alternative collision representations within the same pipeline would help establish the specific value of the Gaussian formulation.

A1. Thank you for this question.

Central advance. Our central advance is to jointly optimize contact anchors and robot key configurations during kinematic retargeting to obtain robot-adapted stable contacts; these contacts greatly improve the quality of the kinematic retargeting results, enabling gradient-based local optimization, which depends on its reference trajectory, to achieve high-fidelity retargeting.

Representative prior contact-refinement methods do not optimize contact anchors and robot key configurations jointly: C2Dex, SPIDER, and Lakshmipathy et al. directly use contacts extracted from the human hand without optimizing the contact anchors, while GenHand targets single-frame grasps and first optimizes the anchors alone before solving for the robot configuration. These methods may force the robot to fit unreachable anchors, producing unreasonable contacts. A new sequential ablation confirms this: refining the anchors first and then solving the robot key configurations with the anchors fixed reaches only 44.6% on DexYCB and 10.3% on HOT3D, below even the raw human anchors (63.4% / 18.9%) and far below joint optimization (78.9% / 32.0%). In our observation, this is because optimizing the anchors alone, unconstrained by the hand's configuration, often moves them far outside the hand's reachable region. In contrast, while optimizing the contact anchors we account for the differences between the human hand and the robot configuration, obtaining robot-adapted stable contact anchors that follow the human contact intent. These anchors are optimized for reachability, collision avoidance, and force closure to obtain high-quality kinematic retargeting results. The table below isolates each design choice.

Dataset  No contact guidance  Raw human anchors  Sequential (anchors first, then configurations)  Joint optimization (ours)
DexYCB  38.3%  63.4%  44.6%  78.9%
HOT3D  4.0%  18.9%  10.3%  32.0%

Success rates averaged over seven hands (Table 1); the sequential variant is new (Sec. 4.2.2 and Appendix G of the revised PDF).

For differentiable retargeting, gradient-based methods are efficient and precise, but contact gradients are unreliable in complex contact and the optimization is local, so they cannot search over contact modes as sampling methods can. Prior methods stabilize the optimization by softening gradients or through annealing curricula (QuasiSim), but do not optimize the input reference with respect to contact. We instead optimize stable contact anchors already in kinematic retargeting, which largely determines the contact mode, so that gradient-based optimization only refines the motion under a good reference, with the anchors remaining its interaction targets, instead of exploring contact modes or making large updates, which gradients do poorly. This is why our gradient-based optimization achieves high precision. This core advance also improves other physics-based retargeting methods (e.g., Contact-aware+SPIDER in the paper), so it is not limited to gradient-based retargeting.

AuthorsOpenReview ↗

Response to Reviewer 4CZr [2/3]

Gaussian formulation. The Gaussian formulation is a supporting contribution: we chose Gaussians because one representation of both appearance and geometry can be built directly and conveniently from video, avoiding two separate representations. This unified representation provides both the appearance needed for tracking and the collision geometry for simulation, with differentiable collision detection. Its specific value is that it improves reconstruction, tracking, and retargeting accuracy over a mesh built from the same frames (table below); it is, however, not irreplaceable in our retargeting pipeline, which supports any representation that provides differentiable collision detection. We have revised the contribution statement accordingly (Sec. 1 of the revised PDF): robot-adapted stable contacts as the central advance, and the Gaussian formulation as a supporting contribution.

To show the compatibility of our method and the specific value of the Gaussian formulation, on the DexYCB ablation subset with the Sharpa and LEAP hands (5 sequences × 2 hands, five seeds), from the same real RGB-D frames, we reconstruct both Gaussians and a mesh (made watertight and colored), track each with a suitable tracker (FoundationPose with the same eight views for the mesh), and run our retargeting pipeline. We also add a group in which the mesh uses the same trajectories as the Gaussians, to isolate the effect of the collision representation on the retargeting pipeline.

Routenear-surface SDF error vs dataset meshPSNR  tracking (center / rotation) retargeting vs tracked reference (E_pos / E_rot on successes / success rate)retargeting vs GT (E_pos / E_rot on successes / success rate)
Our Gaussians2.77 mm20.21 dB  3.78 mm / 3.14° 1.20 mm / 0.30° / 80%2.05 mm / 1.63° / 60%
Our tracking + reconstructed-mesh collision3.75 mm20.21 dB  3.78 mm / 3.14° 1.52 mm / 0.31° / 80%2.31 mm / 1.63° / 60%
Reconstructed mesh + FoundationPose + mesh collision3.75 mm16.80 dB  6.20 mm / 5.95° 1.73 mm / 0.41° / 80%2.90 mm / 3.08° / 60%

Comparing rows 1 and 2, switching only the collision representation from Gaussians to mesh causes only a small decrease in retargeting. Comparing rows 1 and 3, our Gaussians provide a better visual representation during reconstruction (PSNR 20.21 vs 16.80 dB), with better tracking quality (3.78 mm / 3.14° vs 6.20 mm / 5.95°).

On the same real RGB-D data, we also compare with other Gaussian constructions. Our formulation gives the most accurate collision geometry, and its retargeting performance is better than ContactGaussian-WM and far better than spherical 3DGS:

Gaussian constructionnear-surface SDF error vs dataset meshtracking (center / rotation)  retargeting vs tracked reference (E_pos / E_rot on successes / success rate)retargeting vs GT (E_pos / E_rot on successes / success rate)
Ours2.77 mm3.78 mm / 3.14°  1.20 mm / 0.30° / 80%2.05 mm / 1.63° / 60%
ContactGaussian-WM3.26 mm3.70 mm / 3.28°  1.51 mm / 0.42° / 80%3.22 mm / 2.47° / 60%
Spherical 3DGS9.43 mm7.90 mm / 7.41°  2.93 mm / 0.69° / 40%2.20 mm / 2.08° / 20%
AuthorsOpenReview ↗

Response to Reviewer 4CZr [3/3]

Q2: Could you share any available analysis of sequences with frequent contact changes or sliding contacts, and any object-trajectory measurements from real-world execution? Such evidence would help assess the scope of the high-fidelity reproduction claim.

A2. Thank you for this suggestion.

Frequent contact changes and sliding. We binned the HOT3D sequences into tertiles by contact-switch rate (contact starts and ends summed over five fingers) and by the fraction of sliding frames (tangential speed above 20 mm/s).

Success (%): DexForge / Contact-aware+SPIDER  low  medium  high
contact-switch rate (none / 0.2–1.4 / ≥ 1.5 per second)  44.2 / 1.3  37.6 / 0.4  20.9 / 0.0
sliding-frame ratio (≤ 45% / 45–56% / > 56%)  51.0 / 1.0  20.4 / 0.2  33.2 / 0.6

Success decreases as contact switching becomes more frequent, but the stable-anchor formulation still handles these sequences: DexForge stays far above the baseline in every tertile, reaching 20.9% versus 0.0% in the tertile with the most frequent contact changes. Sliding shows no monotonic relation; slow and moderate sliding is segmented into consecutive anchor intervals.

To evaluate the stable contact anchors on the hardest sequences, we took the 10 HOT3D sequences with the most frequent contact changes (3.0 to 9.0 contact switches per second; LEAP and Sharpa hands, five seeds) and compared the full method with a variant that removes contact guidance from both the kinematic and dynamics stages:

10 most frequent contact-switch sequences  10 mm / 0.1 rad (E_pos / E_rot / success rate)  20 mm / 0.2 rad (E_pos / E_rot / success rate)
Full method  5.32 mm / 1.92° / 20%  8.70 mm / 2.56° / 40%
Full method (without contact guidance)  — / — / 0%  15.58 mm / 4.13° / 25%

These results show that, even on sequences with frequent contact switching, our stable anchors still bring a significant improvement rather than limiting performance, since each finger can have different anchors in different contact intervals and can thus represent frequent contact switching.

Real-world object trajectories. Because motion-capture markers on the object would affect the grasp, we measured the object trajectories during real-robot execution with an additional calibrated RealSense D456 camera and FoundationPose++, and added Contact-aware+SPIDER as a baseline under the same open-loop replay (five objects, 10 trials each). Errors are against the dataset ground truth and defined on successful trials.

Criterion  DexForge (E_pos / E_rot / success rate)  Contact-aware+SPIDER (E_pos / E_rot / success rate)
grasp-and-hold (paper)  — / — / 80%  — / — / 64%
20 mm / 0.2 rad  13.47 mm / 7.45° / 38%  17.14 mm / 8.78° / 14%
40 mm / 0.4 rad  23.31 mm / 12.14° / 74%  27.62 mm / 16.78° / 44%

DexForge outperforms the baseline at every criterion. These results are now in Sec. 4.3 (Table R1) and Appendix J of the revised PDF.

We hope these results address your concerns; if so, we would be grateful if you would consider updating your assessment.

Reviewer_nv2wOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

DexForge combines Gaussian object reconstruction, robot-adapted contact optimization, and differentiable dynamics to retarget human demonstrations. Evaluation spans 130 sequences and seven robotic hands, with additional MuJoCo transfer and real-robot grasp-and-hold experiments.

Critical strengths
  1. Joint contact and robot-configuration optimization addresses morphology mismatch directly. Table 1 supports the contributions of both contact guidance and refinement across different hands.
  2. With shared contact guidance, success improves over SPIDER from 50.6% to 77.4% on DexYCB and 0.6% to 34.9% on HOT3D. Appendix H also examines runtime and control-knot spacing, strengthening the optimizer comparison.
Critical weaknesses
  1. Sec. 4.2.1 permits reconstructed references for evaluation, leaving total reconstruction-plus-retargeting error unclear. HOT3D reconstruction error is already 12.23 mm, above the 10 mm success threshold, so fidelity against dataset ground truth matters.
  2. Real-world success measures five-second grasp-and-hold on five objects, without object tracking or a baseline. This provides limited evidence for transferring precise in-hand motion.
Questions for authors

Are reconstructed-input success rates scored against dataset ground truth or reconstructed poses? Please provide success rates and tracking errors against dataset ground truth for these runs.

3 comments
AuthorsOpenReview ↗

Response to Reviewer nv2w

Thank you for your valuable comments. We respond to your questions in turn and hope our responses address your concerns; if any remain, we welcome further discussion. In the revised PDF, changes responding to your comments are marked in orange (changes shared with other reviewers in teal): Sec. 4.2.1, Sec. 4.2.3 (Table 3), Sec. 4.3 (Table R1), and Appendices J and N (Table R9).

Q1: Are reconstructed-input success rates scored against dataset ground truth or reconstructed poses? Please provide success rates and tracking errors against dataset ground truth for these runs.

A1. Thank you for this valuable question. In the paper, reconstructed-input runs are scored against the reconstructed poses. The GT-input results, which carry our main comparison, are already scored against the dataset ground truth. On DexYCB, the reconstruction is nearly identical to the ground truth (0.40 mm / 0.46°, Table 5), so the retargeting success rate and pose errors remain almost unchanged when scored against ground truth (50 sequences × 7 hands, same runs scored both ways: success 74.1% vs 73.5%, E_pos 3.24 vs 3.28 mm, E_rot 0.97° vs 0.95°). HOT3D has only three head-mounted views and the objects are manipulated in hand, which makes reconstruction harder. We therefore additionally report, for HOT3D, the retargeting success rate and errors against the dataset ground truth (errors are defined on successful trajectories), under the paper's retargeting criterion (10 mm / 0.1 rad, Sec. 4.2.1) and its transfer criterion (20 mm / 0.2 rad, Sec. 4.3, Table 4):

HOT3D, reconstructed input  E_pos  E_rot  Success rate
10 mm / 0.1 rad, scored against reconstructed poses (Table 3)  5.79 mm  1.68°  31.2%
10 mm / 0.1 rad, scored against dataset ground truth  8.76 mm  4.34°  11.7%
20 mm / 0.2 rad, scored against reconstructed poses  8.15 mm  2.24°  53.5%
20 mm / 0.2 rad, scored against dataset ground truth  15.24 mm  5.91°  51.2%

The drop on HOT3D comes from reconstruction rather than retargeting: the mean reconstruction error (12.23 mm) already exceeds the 10 mm threshold. With perfect reconstruction (ground-truth input), the same pipeline reaches 34.9%, showing that the drop mainly comes from reconstruction; Contact-aware+SPIDER reaches only 0.6% even with ground-truth input, so our 11.7% with reconstructed input, scored against ground truth, is still about 20 times higher. At 10 mm, the reconstruction error alone is enough to fail a trajectory. Under the paper's 20 mm / 0.2 rad transfer criterion, which exceeds the reconstruction error, success against ground truth is 51.2%, only about 2 points below the 53.5% against the reconstructed poses, showing that once the reconstruction offset is tolerated, the retargeted motion reproduces the real object motion as well. We have added these results to Table 3 and Appendix N (Table R9) and stated the scoring reference in Sec. 4.2.1 of the revised PDF.

W2: Real-world success measures five-second grasp-and-hold on five objects, without object tracking or a baseline. This provides limited evidence for transferring precise in-hand motion.

A2. Thank you for this suggestion. Because motion-capture markers on the object would affect the grasp, we measured the object trajectories during real-robot execution with an additional calibrated RealSense D456 camera and FoundationPose++, and added Contact-aware+SPIDER as a baseline under the same open-loop replay (five objects, 10 trials each). Errors are against the dataset ground truth and defined on successful trials.

Criterion  DexForge (E_pos / E_rot / success rate)  Contact-aware+SPIDER (E_pos / E_rot / success rate)
grasp-and-hold (paper)  — / — / 80%  — / — / 64%
20 mm / 0.2 rad  13.47 mm / 7.45° / 38%  17.14 mm / 8.78° / 14%
40 mm / 0.4 rad  23.31 mm / 12.14° / 74%  27.62 mm / 16.78° / 44%

DexForge outperforms the baseline at every criterion, with lower object pose errors and higher success rates in open-loop real-robot execution. These results are now in Sec. 4.3 (Table R1) and Appendix J of the revised PDF.

We hope these results address your concerns; we are glad to clarify further.

As the author–reviewer discussion ends on November 18, we would like to kindly check whether our responses have addressed your concerns. We would be glad to clarify anything that remains unclear.

Dear Reviewer,

We would like to kindly follow up on our previous responses, as the author–reviewer discussion period is coming to an end today, November 18.

We would greatly appreciate it if you could let us know whether our responses have adequately addressed your concerns. If any issues remain unclear or require further clarification, we would be more than happy to provide additional explanations.

Thank you very much for your time and consideration.

Reviewer_XenFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes DexForge, a pipeline for retargeting human hand-object demonstrations to dexterous robotic hands. The method combines Gaussian-based object reconstruction, contact-aware kinematic retargeting, and differentiable simulation for dynamics refinement. The authors evaluate the method on multiple robotic hands and demonstrate improved retargeting accuracy and success rates compared to existing approaches.

Critical strengths
  • The paper addresses an important problem of transferring human demonstrations to dexterous robotic hands
  • The evaluation covers multiple robotic hands and demonstrates improvements over the baselines
Critical weaknesses
  • The method is limited to single-hand manipulation and is evaluated on relatively simple scenes. It is unclear how well it would work in cluttered environments, under heavy occlusions, or with rapid movements. For the pipeline to be scalable to unconstrained egocentric videos, an evaluation on more challenging and realistic scenes would be useful.
  • The method requires access to the manipulated objects for accurate reconstruction and tracking. This significantly limits its applicability to unconstrained videos. Can the method use objects reconstructed by SAM3D or RecGen? Alternatively, can objects be reconstructed directly from the input RGB-D sequences? How would occlusions affect reconstruction and tracking?
  • The improved geometric accuracy is demonstrated on a synthetic example with ground-truth renderings. However, real RGB-D sequences contain noisy depth and occlusions. The impact of the proposed representation on real-world reconstruction quality and final retargeting performance is unclear.
  • The method assumes access to depth information, but its robustness to noisy depth is unclear. Could the pipeline work with RGB-only videos using recent monocular depth estimation methods such as DA3 or MoGe-3?
Questions for authors
  1. How does the method perform on cluttered scenes with heavy occlusions, rapid movements, or bimanual manipulation? Scenes from Ego4D could be a good example.
  2. Can the method use objects reconstructed by SAM3D or RecGen, or reconstruct objects directly from the input RGB-D sequences?
  3. How much does the improved Gaussian representation contribute to reconstruction quality on real data and final retargeting performance?
  4. How sensitive is the method to depth noise? Could it work with RGB-only videos using methods such as DA3 or MoGe-3?
4 comments
AuthorsOpenReview ↗

Response to Reviewer XenF [1/2]

Thank you for your valuable comments. We respond to your questions in turn and hope our responses address your concerns; if any remain, we welcome further discussion. In the revised PDF, changes responding to your comments are marked in purple (changes shared with other reviewers in teal): Sec. 4.2.1 and Appendices K (Tables R2-R3), L (Tables R4-R5), M (Table R8) and O (Table R10).

Q1: How does the method perform on cluttered scenes with heavy occlusions, rapid movements, or bimanual manipulation? Scenes from Ego4D could be a good example.

A1. Thank you for this valuable comment. We would like to kindly clarify that the 80 HOT3D sequences in our evaluation are egocentric videos recorded with head-mounted Project Aria devices. We provide the following analyses and notes on rapid motion, occlusion, and clutter, and extend our method to more complex scenes, for your reference.

Rapid motion (HOT3D; tertiles by object speed; in the fastest linear tertile the per-sequence 95th-percentile speed is 967–1674 mm/s). Translational speed does not affect success; fast rotation is harder, but every tertile stays far above the baseline.

Success (%): DexForge / Contact-aware+SPIDER  slow  medium  fast
linear speed  37.2 / 1.0  33.7 / 0.8  33.7 / 0.0
angular speed  47.4 / 1.2  33.2 / 0.6  24.0 / 0.0

Occlusion. By measuring hand occlusion across the HOT3D data, we find that tracking error barely increases up to 25% hand occlusion; at 25–50% it increases by about 2.4 mm / 1.9°.

Clutter. The DexYCB and HOT3D scenes contain multiple objects; the manipulated object is segmented with SAM 2 and tracked with its own model.

Bimanual egocentric manipulation. To show the performance of our method in more complex scenes, we extend it to the V2D Track 3 dataset (head-mounted camera, with Vicon ground truth), using two Sharpa hands with the method unchanged. We evaluate two reconstruction routes: (a) starting from the real object model, completing tracking and retargeting with our pipeline; (b) reconstructing the object with SAM 3D, then tracking and retargeting in the same way. Results on four sequences (episodes 41, 44, and 45: a pot held with both hands; episode 22: a pitcher, which is large and heavy and therefore shows larger errors) are in Tables A and B; visualizations are on the anonymous website (https://dexforge-review.pages.dev/). The results show that, whether starting from the real object model or from a SAM 3D reconstruction, our method achieves good tracking and retargeting in complex scenes.

Table A (scan)Tracking vs Vicon (pos. RMSE / rot. median)Physical retargeting vs reconstructed reference (pos. mean / rot. mean)Physical retargeting vs Vicon (pos. RMSE / rot. median)
416.5 mm / 1.9°2.3 mm / 0.28°7.8 mm / 2.9°
445.8 mm / 1.0°1.9 mm / 0.18°6.5 mm / 2.0°
459.7 mm / 1.8°1.7 mm / 0.21°10.6 mm / 2.1°
2213.4 mm / 2.2°13.7 mm / 2.62°23.9 mm / 3.8°
Table B (SAM 3D)Tracking vs Vicon (pos. RMSE / rot. median)Physical retargeting vs reconstructed reference (pos. mean / rot. mean)Physical retargeting vs Vicon (pos. RMSE / rot. median)
4110.2 mm / 2.7°2.1 mm / 0.31°13.2 mm / 3.8°
446.5 mm / 1.1°1.5 mm / 0.23°7.8 mm / 3.1°
4514.7 mm / 3.8°2.1 mm / 0.21°17.3 mm / 4.5°
2216.2 mm / 4.5°15.6 mm / 3.45°28.4 mm / 5.6°

(Ego4D has no 3D hand or object ground truth, so it would allow only a qualitative demonstration; we therefore evaluate on egocentric data with ground truth, namely HOT3D and V2D.)

AuthorsOpenReview ↗

Response to Reviewer XenF [2/2]

Q2: Can the method use objects reconstructed by SAM3D or RecGen, or reconstruct objects directly from the input RGB-D sequences?

A2. Thank you for your suggestion. Yes: as the experiments in A1 show, our method can use a SAM 3D object model in complex scenes and still achieve good retargeting, which shows that it does not have to rely on the real object model and strengthens its applicability to unconstrained videos. We also provide a route that builds the object model directly from the input RGB-D video: we select the less-occluded view images from the input RGB-D video and build our Gaussian representation directly from them. On the DexYCB ablation subset with the Sharpa and LEAP hands (5 sequences × 2 hands, five seeds), we compare this route with the paper's approach of directly accessing the real object model, in terms of object reconstruction, tracking, and retargeting.

DexYCB object modelnear-surface SDF error vs dataset meshtracking on real views (center / rotation)retargeting vs tracked reference (E_pos / E_rot on successes / success rate)retargeting vs GT (E_pos / E_rot on successes / success rate)
reconstructed from RGB-D2.77 mm3.78 mm / 3.14°1.20 mm / 0.30° / 80%2.05 mm / 1.63° / 60%
reconstructed from scan renders0.44 mm0.45 mm / 0.53°1.08 mm / 0.84° / 90%1.17 mm / 1.05° / 90%

The object model built from real RGB-D data is less accurate than the scanned-object route in geometry and tracking and has a lower success rate against ground truth. However, its retargeting relative to the tracked reference is comparable, and it still completes most runs against ground truth, which indicates that the gap comes mainly from the object model and tracking rather than from retargeting itself. In summary, your suggestion strengthens our paper by showing that our pipeline need not rely on access to the real object, and in future work we will explore better reconstruction approaches with fewer dependencies.

Q3: How much does the improved Gaussian representation contribute to reconstruction quality on real data and final retargeting performance?

A3. Thank you for your suggestion. The table below shows that, with all representations built from the same real RGB-D frames (as used for the RGB-D reconstruction in Q2; the same subset, hands and seeds) and in the same pipeline, our formulation gives the most accurate collision geometry; its retargeting performance is better than the ContactGaussian-WM representation and far better than spherical 3DGS.

Representationnear-surface SDF error vs dataset meshtracking on real views (center / rotation)retargeting vs tracked reference (E_pos / E_rot on successes / success rate)retargeting vs GT (E_pos / E_rot on successes / success rate)
Ours2.77 mm3.78 mm / 3.14°1.20 mm / 0.30° / 80%2.05 mm / 1.63° / 60%
ContactGaussian-WM3.26 mm3.70 mm / 3.28°1.51 mm / 0.42° / 80%3.22 mm / 2.47° / 60%
Spherical 3DGS9.43 mm7.90 mm / 7.41°2.93 mm / 0.69° / 40%2.20 mm / 2.08° / 20%

Q4: How sensitive is the method to depth noise? Could it work with RGB-only videos using methods such as DA3 or MoGe-3?

A4. Thank you for your suggestion. Our method uses depth only when building the object model; motion reconstruction (Appendix B) uses RGB only. We therefore designed the following experiments on the same DexYCB ablation subset with the Sharpa and LEAP hands (5 sequences × 2 hands, five seeds) to evaluate the effect of depth noise on object reconstruction and retargeting, both with sensor-like noise added in the synthetic rendering setup of the paper and with real sensor noise when reconstructing directly from real RGB-D frames as in Q2. We also evaluate the case without depth information, using depth estimated by DA3. Retargeting success is unchanged at 5 mm noise, and with DA3 the whole pipeline runs from calibrated RGB cameras alone.

Depth used for the object modelnear-surface SDF error vs dataset meshtracking on real views (center / rotation)retargeting vs tracked reference (E_pos / E_rot on successes / success rate)retargeting vs GT (E_pos / E_rot on successes / success rate)
synthetic: clean0.44 mm0.45 mm / 0.53°1.08 mm / 0.84° / 90%1.17 mm / 1.05° / 90%
synthetic: sensor-like noise, 5 mm at 0.5 m0.57 mm0.43 mm / 0.55°1.13 mm / 0.77° / 90%1.18 mm / 1.06° / 90%
synthetic: RGB only (DA3)0.92 mm0.45 mm / 0.53°1.39 mm / 0.96° / 90%1.44 mm / 1.23° / 90%
real frames: RealSense depth2.77 mm3.78 mm / 3.14°1.20 mm / 0.30° / 80%2.05 mm / 1.63° / 60%
real frames: RGB only (DA3)4.31 mm3.71 mm / 3.07°1.03 mm / 0.32° / 90%2.37 mm / 1.88° / 70%

We hope these results address your concerns; if so, we would be grateful if you would consider updating your assessment.

As the author–reviewer discussion ends on November 18, we would like to kindly check whether our responses have addressed your concerns. We would be glad to clarify anything that remains unclear.

Dear Reviewer,

We would like to kindly follow up on our previous responses, as the author–reviewer discussion period is coming to an end today, November 18.

We would greatly appreciate it if you could let us know whether our responses have adequately addressed your concerns. If any issues remain unclear or require further clarification, we would be more than happy to provide additional explanations.

Thank you very much for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.