acceptodds
Under review as a conference paper at ICLR 2027

Grasp-Informed Object Pose Estimation under Hand Occlusion

Abstract

Object pose estimation is essential for extended reality to respond to users' interactions with physical objects. Yet hands naturally obscure parts of an object during interaction, making different poses look alike in egocentric images. We present HIROP, a method for estimating the 6-DoF pose of novel objects by jointly using visual evidence and grasp priors. HIROP uses a few examples of how an object is held to guide pose estimation without retraining the underlying model. At inference, these examples constrain how the object can be oriented relative to the observed hand and where the two should contact. Combined with visual information, these constraints help distinguish visually ambiguous poses. Experiments on HOT3D-Aria and DexYCB show improved pose accuracy over visual and hand–object interaction baselines across two visual implementations, including under severe hand occlusion. On HOT3D-Aria, using eight representative grasps per object and hand side, HIROP improves ADD- accuracy by 18.9 percentage points over the strongest compared baseline.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Object Pose Estimation, XR

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_sbunOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents HIROP, a method for estimating novel objects’ 6D poses under hand occlusion. Given a known object mesh and a small grasp database, it combines visual evidence with grasp-derived orientation and contact constraints throughout proposal, refinement, and selection, improving accuracy on HOT3D-Aria and DexYCB without retraining.

Critical strengths
  1. Practical problem setting. The paper addresses object pose estimation under hand occlusion, a common challenge in real interactions where hand information provides useful constraints beyond object appearance.
  2. Effective use of grasp priors. HIROP integrates grasp information into pose proposal, refinement, and selection without retraining the underlying pose estimator, offering a practical extension to existing methods.
  3. Strong empirical results. Consistent improvements over both object-only and hand–object pose estimation baselines on HOT3D-Aria and DexYCB support the effectiveness of the approach.
Critical weaknesses
  1. Ground-truth wrist localization. HIROP uses GT wrist positions to anchor predicted hands, providing valuable depth information for contact-based correction. Although HOI baselines share this setting, object-only methods lack this cue. Evaluation with fully predicted hand localization is needed to show robustness in practical settings.
  2. Limited evaluation of unseen grasps. Ablations of database size do not evaluate robustness to grasp types absent from the database. Testing non-typical grasps and mismatched grasps (that are not in the distribution) would clarify whether guidance can degrade performance relative to visual-only estimation. Also, similar hand articulations can correspond to different object orientations or contact regions, what happens when grasp retrieval based on hand articulation returns examples whose implied object poses conflict strongly with the visual evidence?
Questions for authors

No questions. I lean towards acceptance if the two issues about experiments are resolved.

5 comments

GT hand wrist translation. Please see the general response.

Unseen grasps analysis.

HIROP treats coarse hand grasp examples as relational constraints that complement visual evidence in each stage, not as poses to reproduce. Grasp examples are inherently coarse; the hand articulation or hand-relative object rotation will be different during runtime. HIROP is designed to utilize only the grasp information needed to resolve visual ambiguity for each stage.

Coarse proposal uses only the hand-relative rotation to generate hypotheses that act as a prior on visual scores. When appearance alone leaves the rotation ambiguous, the rotation candidates include ones that are consistent with how the object is typically held. The grasp probability keeps a small uniform component so it never excludes rotation with strong visual evidence.

Refinement uses contact vertices for depth correction only when a single RGB image is ambiguous. It takes the contacts of the grasp example with the most similar direction, so the correction holds even when the hand articulation differs.

A limitation is that for objects having symmetric affordance(can be grasped alike from several sides such as a simple box), HIROP will not improve accuracy over the visual baseline since hand grasp carries little information about object pose.

We revised the writing to make these explicit, and added the following experiments to support such claims.

We report accuracy against the angle difference between the actual hand-relative object rotation and the closest database grasp (Figure 6). Most frames(89%) are within 60°, and HIROP exceeds its visual-only counterparts under those frames. This shows that when the database stores grasps aligned with an object's affordance, most interactions share a similar hand-relative object rotation. HIROP falls under frames beyond 60° (11%) because estimated rotation is pulled toward a mismatched grasp. These frames include unusual ways of holding the object; we add visual examples of such frames in Figure 7.

Hand articulation mismatch, in contrast, does not degrade accuracy. HIROP does not select a single grasp but weights all stored grasps by articulation similarity and fuses them with visual scores. Accuracy is therefore relatively flat across articulation distance, as shown in Fig 11.

We also report per-object accuracy (Figures 8 and 12). Gains are largest on objects with obvious affordances and often occluded by the hand, such as a wooden spoon or an eraser (up to +77 points), and lowest on objects grasped alike from any side, such as cans or a box, or distinctive and rarely occluded ones, such as a birdhouse(up to -10.6 points).

Thanks. I have one more question: when HIROP falls due to the estimated rotation being pulled toward a mismatched grasp, how does it compare with visual-only counterparts?

Thank you for the follow-up. We agree that reporting this sensitivity is an important part of our method, and we added an unseen grasp analysis on page 10 of the revised main text (Fig. 6 and 7).

Figure 6 compares HIROP with its visual-only counterparts (Co-op for HIROP-c, FoundPose for HIROP-f) by the angle between the true hand-relative object rotation and the closest database grasp (AR_ADD):

Angle to closest database graspFramesHIROP-cCo-opHIROP-fFoundPose
0–10°17%78.556.784.459.4
10–20°26%92.877.991.275.8
20–30°23%90.778.087.070.9
30–45°16%86.274.982.171.8
45–60°7%83.175.177.573.8
60–90°7%76.679.671.574.7
≥ 90°4%43.970.841.761.4

Within 60° (89% of frames), HIROP is above its counterpart by 8.0–21.8 points for HIROP-c and 3.7–25.0 for HIROP-f. At 60–90° it falls about 3 points below, and at ≥ 90°(4% of frames) the gap grows to 26.9 (HIROP-c) and 19.7 (HIROP-f) points.

The small share of frames beyond 60° shows that when the database stores grasps aligned with an object's affordance, most interactions share a similar hand-relative object rotation, consistent with how people naturally interact with the object. Figure 7 shows examples of unusual grasps where HIROP accuracy degrades, each more than 100° from its closest database grasp.

Reviewer_tZo5OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes a method for object pose estimation under hand occlusion. The pose estimation follows a proposal-refinement-ranking paradigm and grasp constrains are incorporated into each stage. The paper achieves SOAT performance on two benchmarks with a training free design, however, a database for onboarding still requires human labor.

Critical strengths
  • SOTA performance on two datasets including highly occluded scenes.
  • Technical soundness of the method. The proposal-refinement-ranking paradigm is a common practice in object pose estimation.
  • Training-free design.
Critical weaknesses
  • The computational cost is not presented in the paper. The retrieval seems expensive, since the comparison time can grow as the database grows in size.
  • HIROP depends on database collected from a close set of objects, so the ability to apply to unseen object is not clearly presented in the experiments or limitations.
  • The importance of grasp information may be overemphasized. In Table 2, the performance on ALL dataset improves over VHPO more clearly than highly occluded scenes. Therefore, I wonder if the improvement is more likely introduced by visual refinement (i.e. MegaPose). In my experience, refinement itself can significantly improve pose accuracy.
  • The writing and presentation are difficult to understand. For example, Table 5 is not clearly presented because the abbreviation is hard to understand. Table 6 has the same problem, as it is difficult to match it to the corresponding part of the paper.
Questions for authors

requests:

  • How to evaluate the help of grasp information in the refinement stage?
2 comments

Computational cost. We add a per-stage latency breakdown to the appendix, comparing with the visual-only counterparts on RTX 2080 Ti. The grasp-aware modules add around 120ms per object, and most runtime comes from the render-and-compare refiner. The highest grasp-aware cost is contact-based depth correction (79- 95 ms), including Gauss-Newton search. Grasp retrieval takes only 10-14ms, as it compares only against the grasps of the same object, which we store 8 per side where accuracy saturates (Fig.5)

Unseen object. Our claim is that HIROP supports novel object pose estimation for objects unseen during training backbone model, same as GigaPose, FoundPose, and Co-op. To do so, we construct visual templates for each object, following previous work, with an additional grasp prior during pre-runtime onboarding. With this information, HIROP can support novel objects without object-specific training, whereas hand-object interaction perception methods cannot (HOISDF, HFL-Net, VPHO). All 33 HOT3D objects in our evaluation are handled this way, without object-specific training, and HIROP improves over its visual-only counterparts on 26 of them (Figures 8 and 13).

Grasp module effectiveness & writing. We admit that our writing, especially for Tables 5 and 6, can be confusing. We provide a clearer version in the revised PDF. Table 5 highlights the effectiveness of grasp-aware modules at each stage, showing ablations using a visual-only design. For example, comparing the third and fourth rows of Table 5 shows the effectiveness of the grasp-informed contact refinement module compared to visual-based refinement, and comparing the fourth and fifth rows shows the effectiveness of the grasp-informed ranking module.

Refinement stage. Iterative visual refinement does actually bring huge improvements in accuracy, independent of our contribution. We claim that (i) HIROP outperforms the visual-only refinement baselines, (ii) HIROP achieves comparable accuracy with training-based baselines following with visual refinement, and (iii) the effectiveness of grasp-aware refinement and selection can be generalized to other baselines. To support the second and third claims, we added experiments in Appendix C (Table 12). After visual refinement, training-based HOI baselines reach accuracy similar to HIROP, which needs no object-specific training (AR_ADD: HFL-Net+Co-op 84.9 and VPHO+Co-op 88.1 vs. HIROP-c 85.0). Adding HIROP’s contact-based depth correction and grasp-aware selection to the same visual refiner improves accuracy on most HOI-refiner pairs, by up to 3.4 points on all frames and 5.7 on highly occluded frames.

Thank you again for your thoughtful review. If any concerns remain or anything is unclear, we would be glad to clarify further.

Reviewer_xyiCOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a training-free add-on to object pose estimators for frames while hand occlusion is present. Each object is from a small grasp database r hand side), storing the hand-relative object pose and contact vertices. It is applied to FoundPose/MegaPose (HIROP-f) and Co-op (HIROP-c), with gains on HOT3D-Aria and DexYCB.

Critical strengths
  • Realistic domain applicability with the hand that causes the occlusion as evidence.
Critical weaknesses
  • oracle hand translation: Sec. 4.1 (l.308–309) says the wrist translation is ground truth. If so, the contact-based depth correction (Eq. 4) anchors object depth to GT. Depth is the dominant error term in ADD, so this would inflate exactly the gains the paper attributes to grasp evidence (Tab. 5). FoundPose, GigaPose and Co-op do not get such information. It seems all main results should use a fully predicted hand (camera-space WiLoR translation), or the paper should show clearly that it is already like this. Tab. 7 is not answering this, because its "Prediction" row may still use the GT wrist.
  • evaluation: Test split (two held-out subjects, 2 FPS) and 292 "severely occluded" frames are made by authors. The extra frames are annotated by aligning the mesh manually (only in App. A), and there is no estimation of accuracy. I guess they are big part of 454-frame highly-occluded subset, and the occlusion claim is depending on this subset. To convince this at least authors should report results with and without these frames, and estimate the annotation error, for example by re-annotate some frames which have GT.
  • in-distribution grasp dataset: The "captured" database is what authors picked by hand from training split of same dataset and same scripted activities. The "generated" one uses G-HOP, which "does not support unseen objects", so DexYCB objects are also seen. In this view, there is no setting for testing the grasps of new user. More interesting to see is 8-grasp FPS result (Fig. 5) together with manual choice, and also accuracy on frames where hand-relative rotation is far from all database entries.
Questions for authors

There is no runtime, even the method is motivated by XR and it runs K=5 hypotheses per hand, five refiner iterations with Gauss–Newton depth search, and DINOv2 scoring.

2 comments

Oracle hand translation. Please see the general response.

Annotation frames evaluation. The annotated process was done independently of the evaluation methods: we annotated the test interaction frames where HOT3D provides no GT object pose and the object is visible enough to annotate. We added an appendix section on annotation analysis, including which objects are annotated and details of our annotation protocol.

In our submitted version, some manually annotated frames were not part of the highly occluded frames subset (see general comment). We agree that reporting their accuracy separately gives a better understanding of our method. The table below splits the highly occluded subset into two parts (AR_ADD / ADD-0.1d / AR_BOP).

MethodLowest 10% visibility per objectAnnotated
Co-op58.9 / 48.9 / 53.160.7 / 42.9 / 55.0
VPHO71.1 / 40.8 / 54.273.6 / 45.0 / 57.5
HIROP-c71.4 / 57.3 / 62.377.3 / 56.0 / 67.1

HIROP is the most accurate on both subsets across all metrics; its ADD-0.1d is higher on the lowest-visibility frames, while others are higher on the annotated frames. HIROP’s gain on the annotated frames comes from their object type. They are mainly small objects that lose tracking more often under hand occlusion (e.g., whiteboard_marker, mouse) and are held in a characteristic way that benefits most from grasp guidance (object-wise analysis in Figure 8).

To estimate annotation accuracy, we re-annotated 100 frames that have GT in HOT3D under the same protocol, each matched to an annotated frame in object and visibility. Overall, AR_ADD: 95.6 / ADD-0.1d: 97.0 / AR_BOP: 88.5, with a median deviation from ground truth of 4.5 mm (3.0% of the object diameter). The accuracy is far above our method on the annotated frames, so annotation error is small relative to the errors of the compared methods.

In-distribution grasp dataset. We claim that the opportunity of using grasp knowledge in our method comes from the affordance of the object, i.e., how people naturally interact with the object, rather than specifically scripted actions or cherry-picked grasp types.

We split the HOT3D[1] by participants, so evaluation settings ensure the testing of the new user’s grasps. Also, HOT3D describes high level instructions without any specific activity description: “subjects were asked to perform typical actions in a kitchen, office, and living room” (Sec. 3), “we asked subjects to naturally interact with the objects” (Appendix D).

We envision a user capturing a few representative grasps during onboarding, and our evaluation approximates this choice per object. Figure 5 varies the database size with FPS starting from the most representative grasp, showing a small decrease with manually picked: 83.8 / 81.6 for HIROP-c / -f (selected: 85.0 / 83.2).

To show how HIROP behaves across different grasps during runtime, Figure 6 reports accuracy against the angle difference between the actual hand-relative object rotation and the closest database grasp. Most frames(89%) are within 60°, and HIROP exceeds its visual-only counterparts under those frames. This also shows that when the database stores affordance-typical grasps, most interactions share a similar hand-relative object rotation. HIROP falls under frames beyond 60° (11%) because estimated rotation is pulled toward a wrong grasp. These frames show unusual ways of holding the object (Figure 7).

We agree that the generated DexYCB database covers objects seen by G-HOP. Hand generation models that support unseen objects (e.g., OpenHOI[2]) are emerging, and HIROP is orthogonal to the choice of generator.

Computational cost. We add a per-stage latency breakdown to the appendix, comparing with the visual-only counterparts on RTX 2080 Ti. The grasp-aware modules add around 120ms per object, and most of the runtime is the render-and-compare refiner. The highest grasp-aware cost is contact-based depth correction (79- 95 ms), including Gauss-Newton search.

[1] P. Banerjee, S. Shkodrani, P. Moulon, S. Hampali, S. Han, F. Zhang, L. Zhang, J. Fountain, E. Miller, S. Basol, R. Newcombe, R. Wang, J. J. Engel, T. Hodaň. HOT3D: Hand and Object Tracking in 3D from Egocentric Multi-View Videos. CVPR 2025.

[2] Z. Zhang, Y. Shi, L. Yang, S. Ni, Q. Ye, J. Wang. OpenHOI: Open-World Hand-Object Interaction Synthesis with Multimodal Large Language Model. NeurIPS 2025 (Oral).

Thanks for providng the thorough rebuttal. It resolves my major concerns about the paper. I will raise my score accordingly.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

We sincerely thank the reviewers for their thoughtful feedback. We uploaded a revised PDF, with changes marked in blue. Here, we clarify the updates in the revised version and address the concern shared by several reviewers.

Revised evaluation subsets. The revised paper (i) adds all annotation frames to the highly occluded subset; some were missed in the occluded frames in the submitted version. It also (ii) excludes the frames where the hand detector finds no hand for fair comparison: HIROP’s hand input is unavailable there, while the HOI baselines use ground-truth hand boxes. This makes all frames 4,402 -> 4,262(-140 w/o hand), highly occluded 454 -> 701(+292 annotated, -45 duplicate or w/o hand). We provide accuracy for annotated data separately in a comment for xyiC.

Updated baselines. While verifying our evaluation code, we fixed a bug in the model-based RGB baselines: the object models' texture coordinates were flipped. We also retrained the hand-object baselines with longer schedules until convergence. This improves the overall baseline accuracy, not ours. To provide a direct comparison with the submitted version, the table below reports the updated accuracy under the submitted evaluation frame subset of Table 2 (AR_ADD/ADD-0.1D/AR_BOP).

MethodAll (4,402)Highly occluded (454)
GigaPose48.0/32.8/36.5 → 62.2/48.1/50.935.6/20.7/28.3 → 48.3/32.4/40.1
FoundPose63.0/47.1/51.5 → 68.6/54.8/58.849.8/33.0/41.9 → 54.9/39.9/47.8
Co-op65.7/53.7/57.2 → 71.5/63.3/65.753.8/43.6/46.8 → 57.2/47.4/51.9
HOISDF46.3/14.0/19.9 → 56.6/23.7/31.239.4/15.4/17.1 → 50.4/18.9/27.5
HFL-Net64.1/37.1/53.6 → 72.5/52.7/63.449.1/23.6/41.6 → 56.1/32.6/48.2
VPHO76.8/50.0/60.6 → 78.3/53.7/64.268.8/39.6/50.8 → 70.7/41.0/53.7
HIROP-f81.0/67.7/69.969.3/51.1/57.8
HIROP-c82.7/72.6/75.270.2/56.4/61.0

On entire frames, HIROP remains the most accurate in every metric. On highly occluded frames, it leads in ADD-0.1d and AR_BOP, while VPHO is on par in AR_ADD. VPHO is trained on HOT3D, including the test objects, whereas HIROP uses only 8 grasp examples.

Oracle hand wrist translation. We use GT wrist position during evaluation, which is the same with the HOI baselines but not the RGB-only methods. We claim two points here:

(i) The hand we assumed is close to what a commercial XR headset can provide. Our hand with GT wrist has a mean keypoint error of 13.5 mm, and the two-view UmeTrack tracker reaches 10.9mm on Quest3[1].

(ii) Across various single-RGB hand prediction methods, HIROP outperforms its visual-only counterpart (Section 4.5, Appendix D). We added new experiments to support these claims, as detailed below.

  • Wrist depth error from a single RGB image (HOT3D-Aria, 7,691 test hands, mm)

Recovering 3D hand position from a single RGB image is challenging because of the intrinsic 2D-to-3D ambiguity and variation in hand size across people. In addition to taking the wrist directly from WiLoR, we refit it to a SAM hand mask with segmentation-based root fitting (CMR [2]). Here, “calibrated” hand size means the ground-truth hand shape assuming a per-user calibration, which commercial headsets already run during onboarding (e.g., Apple Vision Pro). Single RGB estimates are about 20-27 mm median with a heavy tail (p90 70–76 mm).

Hand inputMedian90th pct.
WiLoR (fully predicted)26.675.4
WiLoR + SAM mask fitting23.476.1
WiLoR + SAM mask fitting + calibrated hand size19.670.1
WiLoR + GT wrist (main experiments)0.00.0
  • Stage-wise accuracy with hand input (HIROP-f on HOT3D-Aria, ADD-0.1d on all frames, occluded frames in parentheses). The first row is FoundPose, which uses no hand input.
ProposalRefinementSelectionWiLoR (fully predicted)WiLoR + SAM mask fittingWiLoR + SAM mask fitting + calibrated hand sizeWiLoR + GT wrist (main experiments)
VisualMegaPoseMegaPose score56.1 (39.2)56.1 (39.2)56.1 (39.2)56.1 (39.2)
+ graspMegaPoseMegaPose score60.0 (42.4)58.9 (40.5)58.8 (40.8)58.2 (40.2)
+ grasp+ contact depthMegaPose score59.1 (41.8)59.1 (42.8)59.8 (45.6)63.7 (47.5)
+ grasp+ contact depth+ grasp61.7 (46.1)62.3 (46.8)62.9 (49.2)69.5 (52.5)

HIROP works with a fully predicted hand. Taking the wrist directly from WiLoR, HIROP-f still exceeds FoundPose by 5.6 points overall and 6.9 points on highly occluded samples. A better wrist estimate translates directly into higher accuracy. Refitting the wrist to the hand mask, still without any ground-truth input, raises the margin over FoundPose to 6.2 points (7.6 on highly occluded). Adding a calibrated hand size raises it to 6.8 points (10.0 on highly occluded).

Contact-based depth correction is the stage most sensitive to wrist depth, and it fails on the tail of the wrist error: it gains 5.0 points when the wrist depth error is within 20 mm (47% of frames), close to the 6.2 with the GT wrist, and loses accuracy beyond 40 mm (27%; Figure 9). Its gain grows steadily as the wrist improves (see rows 2-3). On highly occluded samples, it contributes −0.6, 2.3, and 4.8 points as the median wrist error falls from 26.6 to 23.4, and 19.6 for each RGB-based prediction method. HIROP-c is less sensitive to the hand input: with the fully predicted hand, it loses only 1.6 points from the GT wrist(Table 7), which suggests that the Co-op refiner recovers depth with less reliance on the contact cue. Appendix D of the revision gives the detailed analysis.

[1] P. Banerjee et al., “HOT3D: Hand and Object Tracking in 3D from Egocentric Multi-View Videos,” CVPR 2025. (Tab. 2)

[2] X. Chen, Y. Liu, C. Ma, J. Chang, H. Wang, T. Chen, X. Guo, P. Wan, W. Zheng, “Camera-Space Hand Mesh Recovery via Semantic Aggregation and Adaptive 2D-1D Registration,” CVPR 2021.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.