acceptodds
Withdrawn from ICLR 2027

Geo2Dex: Learning Multi-Embodiment Dexterous Retargeting from Human Videos via Feed-Forward 3D Reconstruction

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Learning from human manipulation videos offers a promising path toward scalable robot learning by reducing dependence on costly robot demonstrations. Accurate retargeting is central to this goal, as the morphological gap between human and robotic hands prevents direct reuse of demonstrated motions. In this paper, we present Geo2Dex, a novel two-stage framework that couples feed-forward 3D reconstruction with embodiment-conditioned trajectory prediction to learn retargeting from paired human–robot demonstrations. Geo2Dex retains reconstructed hand motion and latent scene geometry in a shared interaction representation, which guides a structure-conditioned slot decoder to generate wrist and joint trajectories across robotic hands. Comprehensive evaluations on HRDexDB and DexCap demonstrate marked improvements in reconstruction accuracy, retargeting performance, and simulation replay success. Using these trajectories for fine-tuning also substantially improves vision–language–action policy performance in simulation and on real robots.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Human-to-Robot Motion Retargeting, 3D Hand Motion Reconstruction, Visual Geometry Grounded Transformer (VGGT)

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_XiTXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Geo2Dex learns dexterous human-to-robot retargeting from paired demonstrations. A scene-grounded hand reconstruction stage uses 4DVGGT features to recover human hand motion and latent interaction features. A spatiotemporal encoder and embodiment-conditioned slot decoder then predict wrist and joint trajectories for hands with different action dimensions. Training uses temporally aligned human-robot trajectories and recursive rollout fine-tuning. Experiments cover hand reconstruction, trajectory fidelity and simulation replay on HRDexDB, an additional DexCap evaluation, and downstream VLA fine-tuning in simulation and on a real robot.

Critical strengths
  1. Substantial reconstruction and replay gains. The combined pipeline improves scene-relative hand localization and retargeting across two robotic hands. On HRDexDB, absolute hand-joint error falls from 17.58 mm for the strongest listed reconstruction baseline to 8.21 mm (Table 1). Average replay success rises from the best baseline's 48.94% to 69.15% (Table 4), a 20.21 percentage-point improvement. These results go beyond pose reconstruction alone and support the practical value of the combined pipeline.

  2. Downstream utility and component evidence. Table 5 compares Geo2Dex and Dex-Retargeting under matched rendering, sampling, instructions, and policy training settings (Appendix G), with consistent improvements across both hands and all three tasks. The DINOv3 scene-feature substitution and object-context ablations (Tables 7-8) support the importance of the interaction representation. The real-robot results and supplementary demonstrations provide additional, although small-scale, evidence of feasibility.

Critical weaknesses
  1. Test-time state and split protocol are underspecified. Equation (1) conditions on robot-state history through t while predicting a chunk starting at t; Section 3.6 and Appendix E.2 mention recursive generation and qpos-only rollout fine-tuning, but do not specify initialization, wrist-state updates, or whether the reported trajectory errors use recursively generated states or recorded reference states. The 7:1:2 split also lacks a definition of the split unit and how paired demonstrations, overlapping windows, and reconstruction-stage training are separated. These details are needed to determine whether the reported errors measure independent trajectory generation on held-out demonstrations.

  2. The structural decoder's contribution is not isolated. The current comparisons do not distinguish the decoder's benefit from that of paired robot supervision. Tables 7-8 examine scene features and object context, but do not compare the slot/attribute mechanism against a matched supervised predictor, such as the same encoder with embodiment-specific heads, or slots without actuator attributes. This matters because scene-conditioned hand reconstruction and paired human-to-robot learning have precedents (e.g., Hand3R and Human2Robot), making the particular integration and multi-embodiment decoder central to the contribution. The real-robot extension is encouraging but cannot resolve this attribution: Appendix H gives 50 paired demonstrations per task, yet does not explain the retargeter's Revo 2 adaptation/training split or the actual mixing ratio. Table 6 contains only ten trials per task (8/10 and 7/10 versus 10/10), so the breadth and reliability of the benefit remain uncertain.

Overall, the reported gains are promising, but the test-time protocol and contribution attribution currently support a weak rejection. A clarified evaluation protocol and convincing evidence for the shared structural decoder could change this assessment.

Questions for authors
  1. Evaluation protocol. For Tables 3-4, how are the initial robot state and every subsequent wrist/joint state obtained? Are any ground-truth robot states used after initialization, and which predicted chunk steps are executed? Please also specify the split unit, whether all windows/embodiment pairings from the same underlying human demonstration stay together, and whether Stage 1 sees any Stage 2 test demonstrations. This would determine how to interpret the reported generalization and rollout errors.

  2. Decoder contribution and baseline conditions. Under identical paired data, inputs, and training budgets, what evidence distinguishes the structural slot decoder from an embodiment-specific output head or a slot decoder without actuator attributes? A targeted comparison under matched conditions would materially affect the assessment of the decoder's contribution. Please also clarify which reconstruction inputs and refinement stages were used for the retargeting baselines in Tables 3-4.

  3. Real-robot training and evaluation. How was the retargeter adapted to Revo 2, which of the 50 paired demonstrations per task trained each component, and what human/robot mixing ratio and robot-data budget were used for Table 6? Are the reported ten trials from a single trained policy per condition? These details would establish what the downstream gains support about using additional human data.

Reviewer_JKyyOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Geo2Dex, a two-stage pipeline from human RGB video to multi-embodiment dexterous retargeting and VLA fine-tuning. The core idea is geometry-aware human reconstruction plus an embodiment-conditioned slot decoder. Experiments report better reconstruction and simulation than prior methods.

Critical strengths
  1. The paper proposes a end-to-end style solution ratherthan common module-stack pipelines. Learning an implicit mapping may be a better way for multi-embodiment adaptation than chaining many hand-designed stages.
  2. The method is stronger on reconstruction accuracy and trajectory replay success, and it further trains a VLA to show that the retargeted trajectories are useful in downstream application.
Critical weaknesses
  1. The method mainly tracks the hand. The object token is only pose, not shape, and there is no object reconstruction. Contact is used for time alignment, not for modeling how the hand touches the object. Simulation success is also replay accuracy against the ground-truth object path, not task success rate. That makes practical usefulness unclear.
  2. Real-robot VLA only shows R and H+R. There is no H-only result on the real robot, and the mix ratio is not given. This does not fully show that the collected human data helps.
Questions for authors
  1. Without using the ground-truth object path, what is the task success rate after retargeting?
  2. On the real robot, what is H-only, and what mix ratio is used for H+R?
  3. The object token only takes SE(3) pose. How does the model handle irregular object shapes?
  4. If object pose is missing at test time, how much does retargeting drop?
  5. Contact is used for time alignment. Is there any contact loss or contact consistency check on the predicted robot hand?
  6. Real data uses SAM3D for object pose. How sensitive are the results to pose noise from that pipeline?
  7. Simulation success is replay vs ground-truth object motion. Can you also report open-loop grasp/place success without that oracle path?
Reviewer_y13KOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes GeoDex, a unified framework for learning multi-embodiment dexterous retargeting from paired human-robot dataset. GeoDex consists of two stages: (1) Scene-Grounded Hand Reconstruction which leverages 4DVGGT backbone with learned hand queries and camera tokens for reconstructing 3D human hand motion. (2) At retargeting stage, the reconstructed human hand, latent scene features, robot state, object pose are jointly encoded and used to compute robot trajectories via embodiment-conditioned slots. Compared to existing human hand reconstruction models and retargeting methods, Geo2Dex reduces reconstruction error and improves success rate of computed actions.

Critical strengths
  • Leveraging human videos as a source of robot data via retargeting is an important and well-motivated problem.
  • Leveraging spatial understanding of 4DVGGT to improve hand reconstruction is sound.
  • Embodiment-conditioned slot trajectory decoding can handle multiple embodiments.
Critical weaknesses
  • It requires paired human-robot data to train. It is unclear to which extent Geo2Dex's retargeting would generalize for in the wild human videos, as the experiments were evaluated on held-out but same datasets.
  • Experiments are not fully supporting the benefit of proposed pipeline.
    • Retargeting baselines were not trained on paired human-robot data, so the performance gain of Geo2Dex may mainly root from its access to paired data. For instance, Geo2Dex w/o object context is still better than all the baselines (Table 8).
    • Moreover, many robot motions can achieve the same task, while robot trajectory reconstruction metric implicitly assumes the retargeting should mimic the teleoperation dataset.
    • Missing ablations and comparisons. Geo2Dex consists of many components, while it is unclear which components are necessary. Given access to paired human-robot dataset, the simpler learned regressor trained on same pairs could be as competitive as Geo2Dex.
    • It is unclear whether offline evaluation uses ground-truth robot state, or the ones from model's prediction.
    • How is ManipTrans evaluated? It requires a separate RL process.
Questions for authors

Please check weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.