- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposed a pipeline called OmniHOI to estimate hand-object interaction and retarget hand pose to dex-hand. The core idea is using phisical constraint, including contact and occlusion, to refine the interaction and trajectory. The proposed method achieves high success rate while keep a good PA-MPJPE result.
Critical strengths
- The paper studies hand–object interaction in a highly specific setting, and the reported metrics are all oriented toward producing useful training trajectories for robots. The experiments show that the proposed method is both effective and efficient.
- The paper is very clearly written and easy to follow. Although the proposed pipeline is complex and involves multiple models and constraint formulations, it is presented in a well-organized manner.
Critical weaknesses
- The novelty of the paper is not particularly strong. The pipeline is built around the combination of SAM 3D, FoundationPose, and WiLoR, and the main contribution largely consists of a set of constraints, loss terms, and other auxiliary components designed around this nearly standard backbone. I have already seen similar designs in multiple prior works, from RoboWheel (Zhang et al.) to VideoManip (Chen et al.). Claiming a full end-to-end pipeline therefore seems overstated. That said, if the proposed constraints can be shown to transfer to other module combinations, their value would be much clearer and more substantial.
- I have some doubts about the source of the reported gains. As shown in Figure 6, the main failure of competing methods appears to be degraded 3D object reconstructions, rather than missing or incorrectly applied contact or other hand–object physical constraints. This raises two follow-up questions. First, do the improvements come from SAM 3D, or from the constraint terms that the paper emphasizes? Second, are the comparisons fair—and how would the results change if all methods used SAM 3D or Hunyuan3D to obtain the object mesh as a reconstruction prior?
Questions for authors
Can the proposed strategy transfer to other module combinations? For example, if SAM 3D is replaced by Hunyuan3D, and WiLoR by HaMeR or another hand estimator, would the strategy still hold?
4 comments
Reply to Weakness 1
Contribution beyond the perception backbone
We agree that the perception backbone is standard, and we do not claim it as a contribution; our previous comment shows that the corrections carry over when the backbone changes. Our contribution is a set of corrections, each applied at the stage where the evidence for it is available, before the error propagates:
- Reconstruction from image evidence. Object-centric, pixel-aligned hand placement with a clip-level metric unit from the hand's known size; a per-finger occlusion-order loss that separates contact from occlusion; a contact–penetration energy; and a repair for fingers that pass through thin objects, where signed-distance gradients vanish.
- Retargeting from contact geometry. A refinement of the kinematic solution that keeps each demonstrated contact region while removing the penetration and crossings introduced by the robot's different morphology.
- Physics-in-the-loop refinement from dynamics. Training-free CMA-ES over keyframe offsets in rolling horizons, which makes the motion executable.
The order matters: a robot finger retargeted from a penetrating human finger starts inside the object and cannot separate from it in simulation, so the error has to be removed before simulation.
Relation to RoboWheel and VideoManip
We thank the reviewer for pointing to RoboWheel, which we will cite and discuss. As we read it, RoboWheel's collision stage adjusts the hand's translation and then the wrist's rotation and translation against penetration, using the object's TSDF, and a residual RL policy then refines hand and object trajectories; its dexterous-hand retargeting is kinematic with contact-preserving constraints, and its data are evaluated through downstream policy learning. OmniHOI's corrections act on finger articulation, finger by finger, using image evidence (occlusion order) and handling fingers that cross thin objects; it refines the robot trajectory itself against the object trajectory in physics, without training; and it measures execution directly, on five dexterous hands.
VideoManip, a baseline in our paper, reconstructs the interaction and then trains a task-specific policy that has to absorb reconstruction errors, relating hand and object through keypoint depth. Without any per-task training, OmniHOI succeeds on 53% of the clips against 28% for VideoManip (30% with our depth estimator).
On "end-to-end"
We do not claim a video-to-robot pipeline as such as our contribution; as discussed above, such pipelines exist. In the paper, "end-to-end" names the evaluation setting of Sec. 4.3, raw video in and executed robot trajectory out, as opposed to the motion-transfer setting of Sec. 4.2, which starts from motion capture. To avoid the reading that the pipeline itself is the claim, we will call this setting video-to-robot transfer in the revision and state the contribution as the corrections above.
Most of my concerns have been addressed. My comment concerns the reconstruction degradation shown in Fig. 6. In its current form, the figure does not clearly separate improvements attributable to the reconstruction backbone from those arising from the proposed contributions. The authors are encouraged to clarify this distinction in the caption or main text, or by adding a subfigure that contrasts backbone-driven gains with gains from the proposed method. Such a clarification would better identify the source of the reported improvements and would be more consistent with the quantitative results and the accompanying analysis. I would be willing to raise my score if this remaining issue is resolved.
Revising Fig. 6: backbone-driven gains vs. gains from the proposed method
We thank the reviewer for the follow-up and agree that Fig. 6 should make this distinction explicit. In the revision we will change it as follows.
Subfigure. On the same examples, we add two columns: EasyHOI with SAM 3D in place of InstantMesh, and our pipeline with SAM 3D + HaMeR. Each step then changes one thing. EasyHOI → EasyHOI with SAM 3D isolates the backbone; EasyHOI with SAM 3D → ours isolates our method, with the object generator and hand estimator held fixed.
Caption. We will add: "Differences in object shape largely reflect the object generator. Differences in how the hand sits on the object, i.e. contact and penetration, reflect our placement and refinement: with the same SAM 3D + HaMeR backbone, EasyHOI's object improves but its hand still penetrates the object, whereas ours does not."
Main text. Sec. 4.1 will report the same separation quantitatively, on the 1,500 frames of Table 1:
| What changes | CD (cm²) ↓ | IV (cm³) ↓ | |
|---|---|---|---|
| EasyHOI (InstantMesh + HaMeR) | – | 1.711 | 23.66 |
| EasyHOI (SAM 3D + HaMeR) | backbone | 1.173 | 24.06 |
| Ours (SAM 3D + HaMeR) | our pipeline, same backbone | 0.995 | 1.02 |
The backbone improves the object but leaves hand–object intersection unchanged; on the same backbone, our pipeline lowers it from 24.06 to 1.02 cm³.
The same separation holds end to end. On the 60 clips of Table 3 (XHand), we held the perception outputs fixed, including the SAM 3D object, and switched our corrections on and off before physics-in-the-loop refinement (PITL); we will add this ablation to Sec. 4.5:
| Same SAM 3D output | Succ. (%) ↑ | Median max. penetration (mm) ↓ |
|---|---|---|
| PITL only | 23.3 | 20 |
| + reconstruction refinement + PITL | 41.7 | 12 |
| + contact-aware retargeting + PITL | 45.0 | 9 |
| + both + PITL (full pipeline) | 53.3 | 5 |
Without our corrections, the SAM 3D-based pipeline reaches 23.3%, below VideoManip with the same depth estimator (30.0%); with them, 53.3%.
Reply to Weakness 2 and the Question: Source of the gains and other module combinations
This comment answers the second weakness and the question on other module combinations. Our reply to the first weakness (novelty) follows in a separate comment.
Weakness 2. Same object generator and hand estimator, with and without our corrections.
We gave the baselines the object generators the reviewer names wherever their design allows it. EasyHOI runs its object generator as a separate step, so we replaced its InstantMesh with SAM 3D; its hand estimator is HaMeR. FollowMyHold already uses Hunyuan3D-2 and HaMeR and builds its geometric guidance into Hunyuan3D-2's sampling, so its generator cannot be swapped; instead, we ran our pipeline with Hunyuan3D-2 and HaMeR. iHOI predicts the object as an implicit field and has no separate generator. Results on the 1,500 frames of Table 1:
Question. Other module combinations.
With Hunyuan3D-2 in place of SAM 3D and HaMeR in place of WiLoR, and other settings unchanged from the paper, our pipeline stays ahead of every Table 1 baseline on every reconstruction metric, overall (table above) and on each of the three datasets. End to end, on the 60 clips of Table 3:
†Re-run with MoGe-3, the depth estimator our pipeline uses.
Weaker modules lower our success, consistent with our failure analysis, where 14 of 28 failures originate in perception (Appendix F). In reconstruction and end to end alike, every combination stays ahead of all baselines: the corrections carry over to other module combinations.