- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents HIROP, a method for estimating novel objects’ 6D poses under hand occlusion. Given a known object mesh and a small grasp database, it combines visual evidence with grasp-derived orientation and contact constraints throughout proposal, refinement, and selection, improving accuracy on HOT3D-Aria and DexYCB without retraining.
Critical strengths
- Practical problem setting. The paper addresses object pose estimation under hand occlusion, a common challenge in real interactions where hand information provides useful constraints beyond object appearance.
- Effective use of grasp priors. HIROP integrates grasp information into pose proposal, refinement, and selection without retraining the underlying pose estimator, offering a practical extension to existing methods.
- Strong empirical results. Consistent improvements over both object-only and hand–object pose estimation baselines on HOT3D-Aria and DexYCB support the effectiveness of the approach.
Critical weaknesses
- Ground-truth wrist localization. HIROP uses GT wrist positions to anchor predicted hands, providing valuable depth information for contact-based correction. Although HOI baselines share this setting, object-only methods lack this cue. Evaluation with fully predicted hand localization is needed to show robustness in practical settings.
- Limited evaluation of unseen grasps. Ablations of database size do not evaluate robustness to grasp types absent from the database. Testing non-typical grasps and mismatched grasps (that are not in the distribution) would clarify whether guidance can degrade performance relative to visual-only estimation. Also, similar hand articulations can correspond to different object orientations or contact regions, what happens when grasp retrieval based on hand articulation returns examples whose implied object poses conflict strongly with the visual evidence?
Questions for authors
No questions. I lean towards acceptance if the two issues about experiments are resolved.
5 comments
Thanks. I have one more question: when HIROP falls due to the estimated rotation being pulled toward a mismatched grasp, how does it compare with visual-only counterparts?
Thank you for the follow-up. We agree that reporting this sensitivity is an important part of our method, and we added an unseen grasp analysis on page 10 of the revised main text (Fig. 6 and 7).
Figure 6 compares HIROP with its visual-only counterparts (Co-op for HIROP-c, FoundPose for HIROP-f) by the angle between the true hand-relative object rotation and the closest database grasp (AR_ADD):
| Angle to closest database grasp | Frames | HIROP-c | Co-op | HIROP-f | FoundPose |
|---|---|---|---|---|---|
| 0–10° | 17% | 78.5 | 56.7 | 84.4 | 59.4 |
| 10–20° | 26% | 92.8 | 77.9 | 91.2 | 75.8 |
| 20–30° | 23% | 90.7 | 78.0 | 87.0 | 70.9 |
| 30–45° | 16% | 86.2 | 74.9 | 82.1 | 71.8 |
| 45–60° | 7% | 83.1 | 75.1 | 77.5 | 73.8 |
| 60–90° | 7% | 76.6 | 79.6 | 71.5 | 74.7 |
| ≥ 90° | 4% | 43.9 | 70.8 | 41.7 | 61.4 |
Within 60° (89% of frames), HIROP is above its counterpart by 8.0–21.8 points for HIROP-c and 3.7–25.0 for HIROP-f. At 60–90° it falls about 3 points below, and at ≥ 90°(4% of frames) the gap grows to 26.9 (HIROP-c) and 19.7 (HIROP-f) points.
The small share of frames beyond 60° shows that when the database stores grasps aligned with an object's affordance, most interactions share a similar hand-relative object rotation, consistent with how people naturally interact with the object. Figure 7 shows examples of unusual grasps where HIROP accuracy degrades, each more than 100° from its closest database grasp.
GT hand wrist translation. Please see the general response.
Unseen grasps analysis.
HIROP treats coarse hand grasp examples as relational constraints that complement visual evidence in each stage, not as poses to reproduce. Grasp examples are inherently coarse; the hand articulation or hand-relative object rotation will be different during runtime. HIROP is designed to utilize only the grasp information needed to resolve visual ambiguity for each stage.
Coarse proposal uses only the hand-relative rotation to generate hypotheses that act as a prior on visual scores. When appearance alone leaves the rotation ambiguous, the rotation candidates include ones that are consistent with how the object is typically held. The grasp probability keeps a small uniform component so it never excludes rotation with strong visual evidence.
Refinement uses contact vertices for depth correction only when a single RGB image is ambiguous. It takes the contacts of the grasp example with the most similar direction, so the correction holds even when the hand articulation differs.
A limitation is that for objects having symmetric affordance(can be grasped alike from several sides such as a simple box), HIROP will not improve accuracy over the visual baseline since hand grasp carries little information about object pose.
We revised the writing to make these explicit, and added the following experiments to support such claims.
We report accuracy against the angle difference between the actual hand-relative object rotation and the closest database grasp (Figure 6). Most frames(89%) are within 60°, and HIROP exceeds its visual-only counterparts under those frames. This shows that when the database stores grasps aligned with an object's affordance, most interactions share a similar hand-relative object rotation. HIROP falls under frames beyond 60° (11%) because estimated rotation is pulled toward a mismatched grasp. These frames include unusual ways of holding the object; we add visual examples of such frames in Figure 7.
Hand articulation mismatch, in contrast, does not degrade accuracy. HIROP does not select a single grasp but weights all stored grasps by articulation similarity and fuses them with visual scores. Accuracy is therefore relatively flat across articulation distance, as shown in Fig 11.
We also report per-object accuracy (Figures 8 and 12). Gains are largest on objects with obvious affordances and often occluded by the hand, such as a wooden spoon or an eraser (up to +77 points), and lowest on objects grasped alike from any side, such as cans or a box, or distinctive and rarely occluded ones, such as a birdhouse(up to -10.6 points).