acceptodds
Under review as a conference paper at ICLR 2027

OmniHOI: Dexterous Hand-Object Interaction from Monocular Human Video

Abstract

Monocular videos of human manipulation provide abundant dexterous demonstrations, yet reconstructing hand-object interaction from a single view and transferring it to robot hands remain difficult, limiting their direct use for robot execution. Prior methods either require task-specific RL training, limiting scalability, or assume clean motion-capture trajectories and thus cannot operate directly on video. We present OmniHOI, a pipeline that turns an RGB video of hand-object interaction into an interaction-faithful trajectory on dexterous hands. The key idea is to enforce physical consistency using the evidence available at each stage: image evidence during reconstruction, contact geometry during retargeting, and dynamics during physics-based refinement. Each stage optimizes the corresponding representation directly, correcting errors before they propagate downstream or must be absorbed by a learned policy. Across 150 motion-capture trajectories transferred to each of five dexterous hands with 6 to 22 DoF, we achieve 39–89% success, compared with at most 31% for prior transfer methods. On 60 monocular video clips, we achieve 53% success, compared with 28% for the best prior video-to-robot pipeline. Its trajectories also execute on a real bimanual robot across diverse tasks.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Dexterous manipulation, Learning from human videos, Hand-object interaction reconstruction, Human-to-robot transfer, Physics-based trajectory optimization

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_sps3OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposed a pipeline called OmniHOI to estimate hand-object interaction and retarget hand pose to dex-hand. The core idea is using phisical constraint, including contact and occlusion, to refine the interaction and trajectory. The proposed method achieves high success rate while keep a good PA-MPJPE result.

Critical strengths
  1. The paper studies hand–object interaction in a highly specific setting, and the reported metrics are all oriented toward producing useful training trajectories for robots. The experiments show that the proposed method is both effective and efficient.
  2. The paper is very clearly written and easy to follow. Although the proposed pipeline is complex and involves multiple models and constraint formulations, it is presented in a well-organized manner.
Critical weaknesses
  1. The novelty of the paper is not particularly strong. The pipeline is built around the combination of SAM 3D, FoundationPose, and WiLoR, and the main contribution largely consists of a set of constraints, loss terms, and other auxiliary components designed around this nearly standard backbone. I have already seen similar designs in multiple prior works, from RoboWheel (Zhang et al.) to VideoManip (Chen et al.). Claiming a full end-to-end pipeline therefore seems overstated. That said, if the proposed constraints can be shown to transfer to other module combinations, their value would be much clearer and more substantial.
  2. I have some doubts about the source of the reported gains. As shown in Figure 6, the main failure of competing methods appears to be degraded 3D object reconstructions, rather than missing or incorrectly applied contact or other hand–object physical constraints. This raises two follow-up questions. First, do the improvements come from SAM 3D, or from the constraint terms that the paper emphasizes? Second, are the comparisons fair—and how would the results change if all methods used SAM 3D or Hunyuan3D to obtain the object mesh as a reconstruction prior?
Questions for authors

Can the proposed strategy transfer to other module combinations? For example, if SAM 3D is replaced by Hunyuan3D, and WiLoR by HaMeR or another hand estimator, would the strategy still hold?

4 comments
AuthorsOpenReview ↗

Reply to Weakness 2 and the Question: Source of the gains and other module combinations

This comment answers the second weakness and the question on other module combinations. Our reply to the first weakness (novelty) follows in a separate comment.

Weakness 2. Same object generator and hand estimator, with and without our corrections.

We gave the baselines the object generators the reviewer names wherever their design allows it. EasyHOI runs its object generator as a separate step, so we replaced its InstantMesh with SAM 3D; its hand estimator is HaMeR. FollowMyHold already uses Hunyuan3D-2 and HaMeR and builds its geometric guidance into Hunyuan3D-2's sampling, so its generator cannot be swapped; instead, we ran our pipeline with Hunyuan3D-2 and HaMeR. iHOI predicts the object as an implicit field and has no separate generator. Results on the 1,500 frames of Table 1:

Object + handMethodF5 ↑F10 ↑CD (cm²) ↓PA-MPVPE (mm) ↓IV (cm³) ↓
InstantMesh + HaMeREasyHOI (Table 1)0.2190.3891.71114.6323.66
SAM 3D + HaMeREasyHOI0.2700.4571.17313.9724.06
SAM 3D + HaMeROurs0.3640.5620.9957.241.02
Hunyuan3D-2 + HaMeRFollowMyHold (Table 1)0.2010.3601.7738.9219.73
Hunyuan3D-2 + HaMeROurs0.3160.5181.0527.230.58
  • SAM 3D does improve the object, as the reviewer suspects. EasyHOI's Chamfer distance drops from 1.71 to 1.17 cm² with SAM 3D.
  • It does not fix how the hand sits on the object. EasyHOI's intersection volume stays at 24 cm³ and its hand error barely changes. With the same two modules, ours reaches 1.02 cm³ against EasyHOI's 24.06, and 0.58 cm³ against FollowMyHold's 19.73. These hand–object terms are what our corrections act on. Our object metrics also reflect the object's metric scale and pose registration (Sec. 3.1), so we attribute the hand–object gap, rather than the object gap, to the corrections.
  • Hand–object placement, measured directly. On the same frames, the hand's translation error relative to the object is 65.7 mm for ours against 111.7–301.2 mm for the three baselines, and joint error in the object's frame is 81.5 mm against 87.0–206.3 mm. The baselines' failures in Fig. 6 thus include where the hand sits, not only the object's shape.

Question. Other module combinations.

With Hunyuan3D-2 in place of SAM 3D and HaMeR in place of WiLoR, and other settings unchanged from the paper, our pipeline stays ahead of every Table 1 baseline on every reconstruction metric, overall (table above) and on each of the three datasets. End to end, on the 60 clips of Table 3:

Ours: SAM 3D + WiLoR (paper)Ours: SAM 3D + HaMeROurs: Hunyuan3D-2 + HaMeRVideoManip†Do as I Do†
Succ. (%) ↑5342383028

†Re-run with MoGe-3, the depth estimator our pipeline uses.

Weaker modules lower our success, consistent with our failure analysis, where 14 of 28 failures originate in perception (Appendix F). In reconstruction and end to end alike, every combination stays ahead of all baselines: the corrections carry over to other module combinations.

AuthorsOpenReview ↗

Reply to Weakness 1

Contribution beyond the perception backbone

We agree that the perception backbone is standard, and we do not claim it as a contribution; our previous comment shows that the corrections carry over when the backbone changes. Our contribution is a set of corrections, each applied at the stage where the evidence for it is available, before the error propagates:

  • Reconstruction from image evidence. Object-centric, pixel-aligned hand placement with a clip-level metric unit from the hand's known size; a per-finger occlusion-order loss that separates contact from occlusion; a contact–penetration energy; and a repair for fingers that pass through thin objects, where signed-distance gradients vanish.
  • Retargeting from contact geometry. A refinement of the kinematic solution that keeps each demonstrated contact region while removing the penetration and crossings introduced by the robot's different morphology.
  • Physics-in-the-loop refinement from dynamics. Training-free CMA-ES over keyframe offsets in rolling horizons, which makes the motion executable.

The order matters: a robot finger retargeted from a penetrating human finger starts inside the object and cannot separate from it in simulation, so the error has to be removed before simulation.

Relation to RoboWheel and VideoManip

We thank the reviewer for pointing to RoboWheel, which we will cite and discuss. As we read it, RoboWheel's collision stage adjusts the hand's translation and then the wrist's rotation and translation against penetration, using the object's TSDF, and a residual RL policy then refines hand and object trajectories; its dexterous-hand retargeting is kinematic with contact-preserving constraints, and its data are evaluated through downstream policy learning. OmniHOI's corrections act on finger articulation, finger by finger, using image evidence (occlusion order) and handling fingers that cross thin objects; it refines the robot trajectory itself against the object trajectory in physics, without training; and it measures execution directly, on five dexterous hands.

VideoManip, a baseline in our paper, reconstructs the interaction and then trains a task-specific policy that has to absorb reconstruction errors, relating hand and object through keypoint depth. Without any per-task training, OmniHOI succeeds on 53% of the clips against 28% for VideoManip (30% with our depth estimator).

On "end-to-end"

We do not claim a video-to-robot pipeline as such as our contribution; as discussed above, such pipelines exist. In the paper, "end-to-end" names the evaluation setting of Sec. 4.3, raw video in and executed robot trajectory out, as opposed to the motion-transfer setting of Sec. 4.2, which starts from motion capture. To avoid the reading that the pipeline itself is the claim, we will call this setting video-to-robot transfer in the revision and state the contribution as the corrections above.

Most of my concerns have been addressed. My comment concerns the reconstruction degradation shown in Fig. 6. In its current form, the figure does not clearly separate improvements attributable to the reconstruction backbone from those arising from the proposed contributions. The authors are encouraged to clarify this distinction in the caption or main text, or by adding a subfigure that contrasts backbone-driven gains with gains from the proposed method. Such a clarification would better identify the source of the reported improvements and would be more consistent with the quantitative results and the accompanying analysis. I would be willing to raise my score if this remaining issue is resolved.

AuthorsOpenReview ↗

Revising Fig. 6: backbone-driven gains vs. gains from the proposed method

We thank the reviewer for the follow-up and agree that Fig. 6 should make this distinction explicit. In the revision we will change it as follows.

Subfigure. On the same examples, we add two columns: EasyHOI with SAM 3D in place of InstantMesh, and our pipeline with SAM 3D + HaMeR. Each step then changes one thing. EasyHOI → EasyHOI with SAM 3D isolates the backbone; EasyHOI with SAM 3D → ours isolates our method, with the object generator and hand estimator held fixed.

Caption. We will add: "Differences in object shape largely reflect the object generator. Differences in how the hand sits on the object, i.e. contact and penetration, reflect our placement and refinement: with the same SAM 3D + HaMeR backbone, EasyHOI's object improves but its hand still penetrates the object, whereas ours does not."

Main text. Sec. 4.1 will report the same separation quantitatively, on the 1,500 frames of Table 1:

What changesCD (cm²) ↓IV (cm³) ↓
EasyHOI (InstantMesh + HaMeR)–1.71123.66
EasyHOI (SAM 3D + HaMeR)backbone1.17324.06
Ours (SAM 3D + HaMeR)our pipeline, same backbone0.9951.02

The backbone improves the object but leaves hand–object intersection unchanged; on the same backbone, our pipeline lowers it from 24.06 to 1.02 cm³.

The same separation holds end to end. On the 60 clips of Table 3 (XHand), we held the perception outputs fixed, including the SAM 3D object, and switched our corrections on and off before physics-in-the-loop refinement (PITL); we will add this ablation to Sec. 4.5:

Same SAM 3D outputSucc. (%) ↑Median max. penetration (mm) ↓
PITL only23.320
+ reconstruction refinement + PITL41.712
+ contact-aware retargeting + PITL45.09
+ both + PITL (full pipeline)53.35

Without our corrections, the SAM 3D-based pipeline reaches 23.3%, below VideoManip with the same depth estimator (30.0%); with them, 53.3%.

Reviewer_U4HmOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

OmniHOI converts monocular human manipulation videos into trajectories for dexterous robot hands. It combines image-based reconstruction refinement, contact-aware retargeting, and physics-based trajectory optimization. The method achieves strong motion-transfer results across five robot hands, improves end-to-end video transfer over the evaluated baselines, and demonstrates execution on a real bimanual robot.

Critical strengths

The paper studies an important and challenging problem. The motivation is clear: even if the reconstructed hand motion looks correct, the robot may still lose contact or fail to hold the object. It makes sense to address these problems at different stages of the pipeline.

The results on five robot hands are strong. It is encouraging that the method works across different hand structures without training a separate policy for each task.

Critical weaknesses

My main concern is that the technical novelty is not clear enough. The method combines several existing perception and optimization tools. There are useful designs in the geometric correction steps, but it is unclear how much they contribute beyond retargeting followed by physics-based optimization.

Table 5 also raises this question. Most of the improvement comes from physics-in-the-loop optimization, while contact refinement brings only small gains and sometimes slightly reduces success. It does help reduce penetration, but the results do not clearly show that correcting contact before simulation is necessary for successful execution.

Questions for authors

Could the authors add an ablation on the full video pipeline, using the same perception outputs and running physics optimization with and without the earlier corrections? The current ablation uses ground-truth hand motion, so it is hard to judge how useful these steps are when reconstruction errors are present.

Could the authors also report task-completion results or success rates over repeated real-world trials? Object-trajectory error alone may not tell us whether the task is actually completed, especially given the false successes caused by penetration discussed in the paper.

1 comment
AuthorsOpenReview ↗

Reply to Question 2 and Weakness

This comment answers both weaknesses and the task-completion part of the second question. The end-to-end ablation requested in the first question and real-world success rates over repeated trials are in progress and will follow in separate comments.

Weakness 1. Contribution beyond retargeting followed by physics-based optimization.

Two of our baselines are exactly this combination: SPIDER retargets and then optimizes by sampling in simulation, and ManipTrans retargets and then trains a residual policy in simulation. On the same benchmark they reach 15–26% (Table 2). Our physics-in-the-loop refinement on the plain kinematic retargeting, without any other correction, reaches 85–88% on the four higher-DoF hands (Table 5); even SPIDER run in its own setting with closed-loop feedback reaches 28–55% on these hands. The physics stage is therefore itself part of our contribution: per-keyframe offsets searched with CMA-ES over rolling horizons, with a terminal term that rejects grasps that are about to fail (Eq. 4).

The geometric corrections target a different error: the hand–object geometry that is handed to simulation. With the same object generator and hand estimator, EasyHOI and FollowMyHold leave 24.1 and 19.7 cm³ of hand–object intersection, against 1.0 and 0.6 cm³ for our pipeline, and our hand placement relative to the object has 41% lower error than the best baseline. With unchanged hyperparameters, the corrections carry over to other perception modules: end-to-end success is 42% with SAM 3D + HaMeR and 38% with Hunyuan3D-2 + HaMeR, against at most 30% for prior video-to-robot pipelines run with the same depth estimator.

Weakness 2. Table 5 under the ATE criterion.

In simulation, a finger that passes through the object interlocks with it, and the object then follows the hand as it would in a grasp (Appendix E). The ATE criterion counts such interlocks as successes, so it cannot credit a step whose purpose is to remove them:

  • Open-loop, 38 of the 51 tasks that contact refinement turns into failures have fingers passing fully through the object before refinement, and 5 after.
  • After physics-in-the-loop refinement, contact refinement halves the median maximum penetration depth (9.17 → 4.73 mm) and removes contacts that interlock rather than grasp (frames with contact: 41.4% → 35.1%), while success changes by −2 to +3 points.

Evidence for this correction is needed for execution when reconstruction errors are present is what the end-to-end ablation of the first question measures; we will post it in the following comment.

Question 2. Task completion beyond trajectory error.

Penetration-aware success. For every replay we record the maximum depth by which the hand penetrates the object, measured in the same way for all methods, and count an ATE success as a failure if this depth reaches τ. On SharpaWave (150 tasks; ManipTrans does not support this hand):

Successes, n (%)ATE onlyτ = 12 mmτ = 6 mmτ = 3 mm
SPIDER29 (19.3)23 (15.3)12 (8.0)2
OmniRetarget30 (20.0)19 (12.7)11 (7.3)5
Ours134 (89.3)129 (86.0)95 (63.3)15
  • Penetration-related false successes occur for every method, but the baselines depend on them more: at τ = 6 mm they make up 59% of SPIDER's successes and 63% of OmniRetarget's, against 29% of ours.
  • The ranking is the same at every τ. At 3 mm nearly all successes of every method are flagged, which reflects the simulator's soft contacts rather than any one method, so we take ≥6 mm as significant penetration.

Where the object ends up. ATE averages over the whole trajectory, so we also measure the object's pose error at the final frame, against its ground-truth final pose (ours, SharpaWave, all 150 tasks):

TACOOakInk2ARCTICAll
Median Position Error / P90 (cm)0.84 / 2.21.18 / 6.81.68 / 11.21.04 / 5.5
Median Rotation Error (rad)0.0600.0570.1810.066
  • The object ends within 5 cm of its target pose in 88% of tasks (within 10 cm in 94%) and within 0.5 rad in 90%. The largest errors come from ARCTIC's articulated objects.

We will add both criteria to the paper alongside the ATE criterion.

Reviewer_CFRLOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents OmniHOI, a training-free pipeline that turns a monocular RGB video of hand-object interaction into a trajectory for a dexterous robot hand. It has three stages: 1) reconstruction: off-the-shelf models (SAM 3, MoGe-3, WiLoR, SAM 3D, FoundationPose) recover the scene, hands, and objects. The hand is placed relative to the object through a pixel-aligned fit, then refined with an occlusion-order loss, a contact and penetration energy, and a repair step for fingers that pass through thin objects. 2) Retargeting: a kinematic retargeting solution is refined to keep the demonstrated contacts while removing penetration. 3) Physics-in-the-loop refinement: per-keyframe offsets are optimized with CMA-ES in MuJoCo over rolling windows, so the simulated object follows the target trajectory. The paper evaluates hand-object reconstruction, motion transfer to five robot hands in simulation, and end-to-end video-to-robot transfer, and shows qualitative real-world demonstrations. OmniHOI outperforms prior methods in all three evaluations, with the largest gains in motion transfer.

Critical strengths
  1. The paper presents a complete, working system with strong results. The pipeline goes from raw video to robot execution without per-task training, and it outperforms prior transfer and video-to-robot methods across five robot hands with very different designs.
  2. The paper provides careful and honest analysis. The failure analysis of both the baselines and OmniHOI is detailed and informative.
Critical weaknesses
  1. The method is optimized and evaluated in the same simulator. The physics refinement step tunes the trajectory in the same MuJoCo scene, with the same objects and physics settings, that is later used to measure success. The ablation shows this step accounts for almost all of the improvement. So the main result shows that the trajectory can be optimized to succeed in a known simulator, not that it would hold up under the physics differences a real robot faces. This also raises two concerns: 1) The comparison may be unfair: the baselines were likely optimized in their own settings and then replayed in this scene, while OmniHOI was optimized in the evaluation scene itself. 2) The real-world results are only seven qualitative demonstrations, with no success rates over repeated trials. Testing under changed physics (e.g., mass or friction) or in another simulator, and reporting real-world success rates would address this.

  2. It is unclear which new components matter. Most of the gains seem to come from the recent foundation models and the simulator optimization rather than from the proposed techniques: 1) In reconstruction, the refinement steps slightly worsen hand pose error while reducing penetration, so the advantage over baselines comes from the initial placement. The hand metrics are also computed after alignment, which hides errors in where the hand sits relative to the object, the very thing the method targets. 2) In transfer, contact-aware retargeting changes success by only a few points once physics refinement is added. 3) In the end-to-end comparison, the baselines fail mainly because an older depth model misjudges the camera, so part of the gap may come from the newer depth model rather than the method. This makes it hard to separate the technical contribution from the system integration.

Questions for authors
  1. In Sections 4.2 and 4.3, is the physics refinement run in the same scene (object meshes, masses, friction) that is used for evaluation? In the end-to-end setting, which object geometry and mass does the optimization use, the reconstructed ones or the ground truth?
  2. How do the optimized trajectories perform under perturbed physics or in a different simulator? Can you report real-world success rates over repeated trials?
  3. Were the baselines, especially SPIDER, run with the same simulation settings and a comparable compute budget? Were their outputs optimized in a different scene before being replayed in yours?
  4. Can you run the video-to-robot baselines with the same camera and depth estimates as OmniHOI (or with ground-truth intrinsics), to separate the method’s contribution from the choice of depth model?
  5. Can you report hand-object relative metrics for reconstruction, such as relative translation error or contact accuracy, since Procrustes-aligned hand errors do not capture placement relative to the object?
  6. Since penetration can create false successes, how many successes for each method (including baselines) involve significant penetration? How do results change under a success criterion that also penalizes penetration?
3 comments
AuthorsOpenReview ↗

Reply to Question 4&5, and Weakness 1&2

Q4. Baselines with our camera and depth estimates

We re-ran both video-to-robot baselines on the same 60 clips with MoGe-3, the camera and depth estimator our pipeline uses:

MethodCamera / depthSucc. (%) ↑Succ.Prog. (%) ↑Trans. (cm) ↓Rot. (°) ↓
VideoManipMoGe-2 (paper)28.349.48.630.8
VideoManipMoGe-330.049.88.230.8
Do as I DoMoGe (paper)25.048.39.131.3
Do as I DoMoGe-328.352.48.234.3
OursMoGe-353.3715.726

With identical estimates, each baseline gains one or two clips and the gap stays above 20 points. We used the shared estimates rather than ground-truth intrinsics because our pipeline also runs on estimated intrinsics. The difference lies in how the estimates are used: we do not take metric scale from the depth model, but fix one unit per clip from the known size of the MANO hand and place the hand by an object-weighted, pixel-aligned fit (Sec. 3.1).

W1. "most of the gains seem to come from the recent foundation models".

We ran our pipeline with the object generator and hand estimator with those of other methods, keeping the rest of the pipeline fixed. Reconstruction is on the frames of Table 1, end-to-end success on the 60 clips:

Object + handMethodPA-MPVPE (mm) ↓IV (cm³) ↓End-to-end Succ. (%) ↑
SAM 3D + WiLoROurs (paper)6.052.7553
SAM 3D + HaMeROurs7.241.0242
Hunyuan3D-2 + HaMeROurs7.230.5838
Hunyuan3D-2 + HaMeRFollowMyHold (Table 1)8.9219.73–

With exactly the modules FollowMyHold is built on, our intersection volume is 34× lower, so this gap does not come from the modules. Every combination of ours stays ahead of all Table 1 baselines on both metrics and, end to end, of both video-to-robot baselines (at most 30% with MoGe-3), though weaker perception lowers our success, consistent with Appendix F (14 of 28 failures originate in perception).

W2. "Separating the technical contribution from system integration."

The foundation models give per-frame, mutually independent estimates: a hand mesh of unknown depth and scale, an object mesh and pose, a depth map. None of them decides whether a finger touches, occludes or penetrates the object, and none yields a trajectory a robot hand can execute. Our contribution is the set of corrections that turns these estimates into an executable interaction, each applied at the stage where its evidence is available; the swaps above show that these corrections carry over to other modules.

Q5. Hand–object relative metrics

On the 1,500 frames of Table 1 we add relative translation (hand position relative to the object, no scale alignment), OA-MPJPE (joint error in the object's frame) and contact precision / recall / F1 at 5 mm against ground-truth contact:

MethodRel. trans., mean / median (mm) ↓OA-MPJPE, mean / median (mm) ↓Contact P / R / F1 ↑
iHOI99.9 / 52.587.0 / 63.10.264 / 0.206 / 0.199
FollowMyHold150.0 / 115.7206.3 / 103.40.103 / 0.146 / 0.093
EasyHOI128.4 / 97.6145.0 / 93.00.173 / 0.278 / 0.177
Ours65.5 / 48.781.5 / 45.00.289 / 0.275 / 0.210

We lead on every metric except contact recall (0.003 below EasyHOI), most clearly in relative translation (41% lower mean than the next best). Contact F1 is low for every method, and our margin there is modest. Errors are computed after scale alignment against standard MANO hand scale. scale

W2. "the advantage over baselines comes from the initial placement".

Procrustes alignment removes exactly the similarity transform that placement estimates, so the "Placement only" row of Table 4 (5.37 mm) is the error of the raw WiLoR mesh. Placement cannot change PA error; as the reviewer suggests, the PA lead comes largely from the hand estimator. With HaMeR, the estimator FollowMyHold uses, our PA-MPVPE is still lower (7.24 / 7.23 vs. 8.92 mm, table above). What placement and refinement target, the hand's position and contact relative to the object, is what the relative metrics above measure. Refinement raises PA error by 0.68 mm because the ground-truth MANO annotations themselves penetrate the object in places (Fig. 5), and pushing the fingers out moves them away from such annotations. In exchange, it lowers intersection volume by a third (4.27 → 2.75 cm³, Table 4), which matters downstream: a robot hand retargeted from a penetrating human hand starts inside the object.

AuthorsOpenReview ↗

Reply to Question 1&2&3 and Weakness 1

This comment answers Q1, the first part of Q2, and Q3, which together address the concern that our method is optimized and evaluated in the same simulator. Real-robot success rates (second part of Q2) will follow.

Q1. Optimization scene vs. evaluation scene

  • Motion transfer (Sec. 4.2): the same scene. Physics-in-the-loop refinement (PITL) runs in the MuJoCo scene used for evaluation, with ground-truth object meshes, catalogued masses and the contact parameters of Table 8. Table 2 alone therefore shows how well a trajectory can be optimized for a known simulator; Q2 below tests what remains when the physics changes.
  • End-to-end (Sec. 4.3): reconstructed geometry and target. PITL optimizes against the reconstructed object, the SAM 3D mesh at the clip's scale, with the reconstructed object trajectory as its target (Eq. 4); the object mass and contact parameters are the same as in evaluation (catalogued mass, Table 8). Success is measured by replaying the result in the ground-truth scene, against the ground-truth object trajectory. Table 3 therefore already evaluates trajectories under a geometry and target they were not optimized for; its failures include objects reconstructed at the wrong size (12 of 28, Appendix F).

Q2 (first part). Perturbed physics and a second simulator

Without any re-optimization, we replayed the final Shadow Hand trajectories of all methods (a) in our MuJoCo scene, with every task replayed under 10 random combinations of object mass (×0.5, 0.75, 1.5, 2), friction (×0.5, 0.75, 1.5) and contact softness (solref / solimp), and (b) in a second physics engine, Isaac Lab:

Success (%), Shadow HandMuJoCo (Paper)MuJoCo, perturbedIsaac Lab
ManipTrans*2626.028.0
SPIDER1522.116.0
OmniRetarget1620.925.3
Ours8561.956.0

*ManipTrans trains a task-specific policy per trajectory; as in Table 2, all its columns cover 50 of the 150 tasks.

  • Our success does drop, to 61.9% under perturbation and 56.0% in Isaac Lab. We take this as confirming the reviewer's point: PITL partly exploits the dynamics of the scene it optimizes in, and part of the margin in Table 2 reflects this. We will add these results and state this limitation in the paper; scoring each CMA-ES candidate under several sampled physics parameters is a natural remedy.
  • The advantage holds under both changes. Our trajectories succeed at least twice as often as every baseline in both settings, while the baselines stay between 15% and 28% across all three columns.

Q3. Baseline settings and compute

Yes: each baseline produces its output in its own pipeline, and that output is then replayed open-loop in our scene with the shared settings of Table 8 (Appendix D): SPIDER’s output is its sequence of commanded joint targets; ManipTrans’s is the joint states its policy actually reached (best of 10 rollouts, as the policy is stochastic); OmniRetarget is kinematic, so its output is a set of keyframes, which we interpolate as for our method. Compute per trajectory (Table 2) is 37 min for SPIDER, 385 min for ManipTrans and 19 min for ours, all measured in wall-clock time on single A100 (for baselines) and on CPU (for ours). So both simulation-based baselines received more compute than our method.

Two further checks address fairness from both sides:

  • Neutral ground. In Isaac Lab, where our trajectories were never optimized, ours reaches 56.0% against at most 28.0% (Q2 above).
  • The baseline's own ground. We ran SPIDER in its own setting, closed-loop, under the same ATE criterion on all 150 tasks per hand:
Success (%)InspireXHandWujiShadowSharpaWave
SPIDER, own setting, closed-loop5855284151
SPIDER, replayed in our scene (Table 2)2626201519
Ours, open-loop in our scene (Table 2)3989868589

Even on its own ground and with closed-loop feedback, SPIDER stays below our open-loop result on four of the five hands. The exception is the 6-DoF Inspire Hand, where feedback can compensate for the hand's limited dexterity and an open-loop trajectory cannot. SPIDER also loses much of its success when moved to our scene, partly from losing feedback: the same effect the reviewer raises for our method, which Q2 measures for ours directly.

AuthorsOpenReview ↗

Reply to Question 6

Q6. Penetration-aware success

For every replay we record the maximum depth by which the hand penetrates the object over the trajectory, measured in the same way for all methods. An ATE success whose maximum penetration reaches τ is then counted as a failure. We report SharpaWave (22 DoF, 150 tasks); ManipTrans does not support this hand; on the hands it does support, it is evaluated on 50 tasks (as in Table 2) and succeeds on only 11–13 of them, too few to break down by penetration depth.

Successes, n (%)ATE onlyτ = 12 mmτ = 6 mmτ = 3 mm
SPIDER29 (19.3)23 (15.3)12 (8.0)2
OmniRetarget30 (20.0)19 (12.7)11 (7.3)5
Ours134 (89.3)129 (86.0)95 (63.3)15
  • Penetration-related false successes occur for every method, but the baselines depend on them more. At τ = 6 mm, 17 of SPIDER's 29 successes (59%) and 19 of OmniRetarget's 30 (63%) involve significant penetration, against 39 of our 134 (29%). Reducing penetration before simulation is the purpose of our contact-aware retargeting: after PITL, it roughly halves the median maximum penetration depth (9.17 → 4.73 mm, Sec. 4.5).
  • The ranking is the same at every τ, and the gap widens. At τ = 6 mm our success rate is 7.9× that of the best baseline, compared with 4.5× under the ATE criterion alone.
  • On the choice of τ. At 3 mm nearly all successful replays of every method are flagged (27 of 29, 25 of 30, 119 of 134). Penetration at this level therefore reflects how the simulator resolves contact, through small interpenetration under its soft contact constraints (Table 8), rather than a failure specific to any method. We treat ≥6 mm as significant penetration and report the full range so that the conclusion does not rest on a single threshold.

We will add this criterion to the paper alongside the ATE criterion.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the rebuttal: concerns, added experiments and results

We thank the reviewers and the AC. Below we summarize the concerns raised, the experiments we added, and their results. All requested experiments are complete, and the revised PDF integrates them. Reviewers U4Hm and sps3 indicated that the new results address their main concerns; we answered sps3’s remaining request on Fig. 6 with a module-matched subfigure.

Concern (reviewers)Experiments addedKey resultIn the revised PDF
Do the geometric corrections matter beyond physics refinement? (U4Hm, CFRL)End-to-end ablation on the 60 video clips, perception outputs held fixedPhysics refinement alone: 23.3%; with our corrections: 53.3%. Contact-aware retargeting alone adds 21.7 points (vs. ≤3 on ground-truth MANO, which carries less contact error)Sec. 4.5, Table 6
Do the gains come from SAM 3D and other foundation models? (sps3, CFRL)Baselines given our backbone; our pipeline with other modules; video baselines with our depth estimatorSame SAM 3D + HaMeR backbone: hand–object intersection 24.1 cm³ (EasyHOI) vs. 1.0 cm³ (ours). With SAM 3D + HaMeR / Hunyuan3D-2 + HaMeR and unchanged hyperparameters, ours reaches 41.7% / 38.3% end to end. Baselines with MoGe-3: 30.0% / 28.3% vs. 53.3%Sec. 4.1, Fig. 6(b), App. H, K
Optimized and evaluated in the same simulator; fairness to baselines (CFRL)Perturbed physics (mass, friction, contact), a second simulator, SPIDER in its own setting, scene clarificationWithout re-optimization: 61.9% under perturbed physics and 56.0% in Isaac Lab, at least 2× every baseline (≤28%). SPIDER in its own setting, closed-loop, stays below our open-loop result on 4 of 5 hands. In the video setting, refinement uses the reconstructed object and is evaluated in the ground-truth sceneApp. D, J; Limitations
No real-world success rates (U4Hm, CFRL)7 real-robot tasks × 10 trials, open-loop, criteria fixed in advance49/70 (70%); failures concentrate in insertion and stacking, which need precise placementSec. 4.4, App. G
Trajectory error may hide penetration-based false successes; is the task completed? (U4Hm, CFRL)Penetration-aware success criterion (τ sweep); object pose at the final frameAt τ = 6 mm (SharpaWave): 63.3% vs. ≤8.0% for baselines; penetration accounts for 59–63% of baseline successes vs. 29% of ours. Object ends within 5 cm of its target in 88% of tasksSec. 4.2, App. I
Procrustes-aligned hand errors hide hand–object placement (CFRL)Hand–object relative metricsRelative translation error 41% lower than the best baseline (65.7 vs. 111.7 mm); best OA-MPJPE and contact F1App. H
Novelty; relation to RoboWheel / VideoManip; "end-to-end" wording (sps3, U4Hm)Clarified contribution; RoboWheel cited and distinguished; wording revisedContribution stated as stage-wise corrections that carry over across perception modules, not as the pipeline itselfSec. 1, 2, 4.3

In short.

  • The gains do not come from the backbone. With the same SAM 3D output, our corrections more than double end-to-end success (23.3% → 53.3%), and without them our pipeline falls below VideoManip (30.0%). On the same reconstruction backbone, EasyHOI and FollowMyHold keep 24× and 34× more hand–object intersection than ours.
  • They do not come from simulator optimization alone. Physics refinement without our corrections reaches 23.3%; SPIDER, which also optimizes in simulation, stays below our results on four of five hands even in its own setting.
  • They hold beyond our simulator. Under perturbed physics and in Isaac Lab, our trajectories still succeed at least twice as often as every baseline, and on a real robot they complete 49 of 70 trials.

We also state the remaining limitations in the revised paper: our margin shrinks when the physics changes (85% → 56–62%), and open-loop execution limits precise real-world placement.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.