Grasp-Informed Object Pose Estimation under Hand Occlusion

Object pose estimation is essential for extended reality to respond to users' interactions with physical objects. Yet hands naturally obscure parts of an object during interaction, making different poses look alike in egocentric images. We present HIROP, a method for estimating the 6-DoF pose of novel objects by jointly using visual evidence and grasp priors. HIROP uses a few examples of how an object is held to guide pose estimation without retraining the underlying model. At inference, these examples constrain how the object can be oriented relative to the observed hand and where the two should contact. Combined with visual information, these constraints help distinguish visually ambiguous poses. Experiments on HOT3D-Aria and DexYCB show improved pose accuracy over visual and hand–object interaction baselines across two visual implementations, including under severe hand occlusion. On HOT3D-Aria, using eight representative grasps per object and hand side, HIROP improves ADD-$0.1d$ accuracy by 18.9 percentage points over the strongest compared baseline.