acceptodds
Withdrawn from ICLR 2027

STEREOHOI: RECOVER METRIC HAND-OBJECT IN- TERACTION FROM EGOCENTRIC STEREO VISION

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reconstructing 3D hand-object interaction (HOI) in physical metric space is impor- tant for embodied AI and robotic manipulation. Existing HOI approaches mostly rely on RGB observations from monocular images or video streams, where the lack of binocular metric scale makes physical hand-object alignment challenging, often necessitating expensive test-time optimization. While stereo cameras provide binocular disparity cues to resolve metric scale, existing stereo networks focus primarily on dense surface depth estimation, which cannot directly predict hand keypoints or infer their corresponding metric depths. In this work, we explore stereo vision for recovering 3D hand-object interaction and present StereoHOI. Rather than recovering metric hand scale through additional fitting to stereo depth estimates, StereoHOI directly recovers metric hands by fusing monocular hand priors extracted from a frozen hand foundation model within an alternating cross- view attention framework. By encoding the monocular hand priors and camera parameters into conditioned special tokens, StereoHOI implicitly captures binoc- ular disparity cues to predict metric wrist translations and pose residuals. This enables placing hands and metric object meshes into a shared metric coordinate frame, simplifying hand-object alignment and supporting video-based interaction analysis. Code will be released to benefit the research community.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
HOI, Embodied AI, Egocentirc Video

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_DruKOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents StereoHOI, a framework for metric hand–object reconstruction from egocentric stereo video. It combines frozen monocular hand priors with cross-view attention to estimate metric hand geometry. Stereo-based object reconstruction and separate trajectory optimization place hands and objects in a shared coordinate frame, improving reconstruction accuracy on the HOT3D benchmark.

Critical strengths
  1. Practical and well-motivated setting. Leveraging calibrated stereo cameras addresses the depth and scale ambiguities that complicate monocular hand–object reconstruction, which is a practical engineering solution.
  2. Effective stereo hand reconstruction. Comparisons with stereo baselines and input-view ablations support the method’s ability to improve both the hand and object.
Critical weaknesses
  1. Physical plausibility is insufficiently enforced and evaluated. Hand and object trajectories are optimized independently, without explicit contact or interpenetration constraints. A shared metric coordinate frame reduces scale inconsistencies but does not guarantee correct contact or prevent penetration and floating. For evaluation, hand-relative Chamfer Distance and qualitative examples provide limited evidence; direct measurements of contact error, penetration, and separation are needed to show the claimed interaction quality.
  2. Generalization across stereo configurations remains untested. The model is trained on Aria and Quest 3 but evaluated only on Aria. Testing on unseen camera configurations, stereo baselines, and focal lengths is needed to establish whether its metric reconstruction capability generalizes beyond the training setups.
  3. The full HOI comparison does not control for input differences. Although Table 1 includes stereo hand baselines, the HOI methods in Table 2 primarily use monocular video, whereas StereoHOI exploits stereo cues for hand localization, object scaling, and tracking. These results therefore do not separate the proposed method’s contribution from the benefit of additional depth information. Full-pipeline comparisons under matched stereo inputs are needed.
Questions for authors
  1. How does the current framework ensure physical plausibility?
  2. How about the generalizability to in-the-wild videos?
  3. Full-pipeline comparisons under matched stereo inputs?
Reviewer_4xWxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper propose StereoHOI for recovering metric 3D hand from calibrated egocentric stereo pair. For each view, the features and MANO prior are taken, and with camera token they go through transformer. From this, the model predicts 2D keypoints, residuals, and the wrist depth in log space. Hands are refined by trust-region temporal optimization. On HOT3D authors report big improvement over monocular and stereo hand baselines and also over monocular HOI, with less runtime.

Critical strengths
  • Stereo camera is already available in Aria, Quest 3 and robot head, and paper shows that frozen monocular hand foundation model can be lifted to metric stereo with not so much in-domain data.
  • "Duplicated Left" control is good, as it separate the real binocular information from just extra capacity.
  • Results are strong on large test set.
  • The description of the method is good.
Critical weaknesses
  • Performance gain: Authors explain monocular CT error >45 mm by "depth-scale ambiguity". But authors' own monocular variant ("w/o Right View", Tab. 3a) already get CS-J 14.13 / CT 11.66 mm, compared to 53–66 mm of WiLoR/HaMeR. So around 40 mm from ~43 mm improvement seems coming from in-domain metric-depth training with known intrinsics, and stereo is adding only about 3 mm. In my opinion abstract and Sec. 1 are overstating the role of binocular cue.
  • Baseline conern: POEM-v2-FT is fine-tuned with joints triangulated from its own 2D predictions, not with HOT3D GT which StereoHOI is using as in App. A.3. UmeTrack is running on different cameras. HaMeR (Tab. 3a, with 108 mm CS-J) is only one experiment which test the premise of Sec. 1 that "depth maps are unreliable", but how the lifting is done is not described.
  • Similarly, HOLD, BIGS, MagicHOI and Do as I Do are monocular methods, but object branch of StereoHOI uses metric depth from FoundationStereo, multi-view SAM3D and FoundationPose. So 5–30× gains on object mostly come from stereo depth and foundation models.
  • Interaction claim is not convincing: Abstract and Sec. 4.3 say there is less hovering and interpenetration and that hand-object alignment is "simplified", but only evidence is Fig. 4.
  • Contribution: The model is combination of HaMeR priors, WiLoR-style residual refinement, alternating attention and GAU head, and video system is assembling exisiting modules. Evaluation is on one dataset and one camera pair only, even the paper claims about headsets, glasses and humanoids.
Questions for authors

Additional question is, what exactly HaMeR† is doing with FoundationStereo depth?

Reviewer_vq1gOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents StereoHOI, a framework for metric 3D hand-object interaction reconstruction from egocentric stereo videos. The method leverages a frozen monocular hand foundation model (HaMeR) to extract hand priors and visual features from both views, which are fused through an alternating cross-view transformer. A dual-branch decoder predicts 2D hand keypoints and refines MANO parameters, including metric wrist translation, using geometric supervision. For video-level reconstruction, the framework integrates SLAM, stereo depth estimation, object reconstruction, and temporal optimization to recover hand and object trajectories in a shared metric coordinate system. Experiments on the HOT3D benchmark demonstrate improvements over existing monocular and stereo baselines in hand reconstruction accuracy, object reconstruction quality, and computational efficiency.

Critical strengths
  1. Well-designed and coherent system: The paper presents a well-designed framework that integrates metric hand reconstruction, object reconstruction, camera pose estimation, and temporal optimization into a unified pipeline. Rather than relying on expensive joint hand-object optimization, the method reconstructs hands and objects independently in a shared metric coordinate system, followed by temporal refinement.

  2. Clear and well-justified motivation: The paper provides a clear motivation by identifying the limitations of existing monocular methods, which suffer from inherent depth-scale ambiguity, and conventional stereo-based reconstruction methods, which relies on stereo depth estimation methods that struggle with articulated hands due to their relatively textureless surfaces and frequent occlusions. The proposed approach addresses these challenges by combining strong monocular hand priors with cross-view stereo information to directly predict metric hand parameters, avoiding the need for explicit depth-based hand fitting.

  3. Strong experimental results with substantial improvements: The paper provides comprehensive quantitative comparisons against a wide range of monocular, stereo, and video-based reconstruction methods. The proposed method consistently achieves strong performance across multiple evaluation metrics, including hand pose accuracy, metric translation accuracy, object reconstruction quality, and temporal tracking.

Critical weaknesses
  1. Limited technical novelty: Although the overall system is well-designed and practically motivated, most of its components are adapted from existing methods. The hand reconstruction network combines pretrained monocular hand priors from HaMeR with an alternating cross-view transformer inspired by prior work, followed by relatively standard MANO parameter regression and geometric supervision. Similarly, the video-level HOI reconstruction pipeline integrates several existing foundation models and optimization techniques. While the combination is effective, the paper introduces relatively limited novel modeling or algorithmic contributions.

  2. Limited improvements from several proposed components: Although the ablation studies demonstrate the benefits of stereo inputs, the improvements brought by several other design choices are relatively modest. For instance, removing the prior token initialization increases CS-J from 11.17 mm to 11.88 mm, while replacing residual parameter prediction with direct decoding increases it to 11.45 mm. Similarly, removing the 2D keypoint branch only increases CS-J to 11.52 mm.

  3. Additional baseline comparison is desired: It's desirable to compare the proposed alternating cross-view transformer architecture with simpler stereo feature fusion strategies, such as concatenating features from both views followed by an MLP or applying standard joint self-attention to the concatenated tokens without alternating intra-view and cross-view attention.

Questions for authors
  1. Contribution of individual components: As mentioned above in the "Weakness" section, several major design brings relatively modest performance gains. Could the authors further clarify the importance of these design choices?

  2. Camera geometry and generalization: The proposed method encodes camera intrinsics and the stereo baseline into a camera token. Could the authors clarify how camera extrinsics, stereo rectification, and coordinate transformations introduced by hand cropping are handled? Furthermore, how well does the model generalize to different stereo camera configurations?

Reviewer_nYbEOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents StereoHOI, a framework for metric hand and hand–object interaction reconstruction from egocentric stereo videos. The hand reconstruction network combines frozen HaMeR features and hand estimates with camera-conditioned tokens and an alternating cross-view transformer. It predicts refined MANO parameters, metric wrist translation, and 2D keypoints. The video pipeline integrates existing camera tracking, stereo depth, object reconstruction, and pose estimation models, followed by sequence-level trajectory optimization. Experiments on HOT3D demonstrate improvements in hand reconstruction and hand–object tracking.

Critical strengths
  1. The paper addresses an important practical problem. Recovering hands and objects in a shared metric coordinate system is useful for understanding manipulation and collecting data for embodied learning.
  2. Adapting strong monocular hand priors to calibrated stereo input is a sensible approach. The separation between articulated hand reconstruction and stereo-based object geometry is well motivated.
  3. The reported results are promising. StereoHOI achieves a camera-space joint error of 11.17 mm and a wrist translation error of 8.72 mm on the evaluated Aria test set. The comparisons include both monocular and stereo baselines, and the ablations examine several architectural and temporal refinement choices.
  4. The paper distinguishes frame-wise hand prediction from subsequent sequence optimization, and the appendix provides useful details about the latter.
Critical weaknesses
  1. The methodological contribution beyond integration of existing components is not sufficiently established. The framework relies on frozen HaMeR features and initial hand estimates, as well as DROID-SLAM, FoundationStereo, MV-SAM3D, and FoundationPose. Using pretrained components is reasonable and does not itself diminish the contribution. However, it makes it particularly important to isolate the benefit of the proposed stereo architecture from that of the underlying models and the availability of calibrated stereo observations. The trainable 24-layer transformer is a substantial module, so my concern is not that too few parameters are trained. Rather, the paper does not sufficiently establish why the proposed design is preferable to simpler ways of combining the same pretrained features and camera information. A controlled comparison with simpler feature concatenation or cross-view fusion, using the same training data and downstream pipeline, would make the architectural contribution more convincing.
  2. The ablations support the design, but leave questions about the source and magnitude of the gains. Removing the right view increases CS-J from 11.17 to 14.13 mm, while duplicating the left view yields 13.76 mm. These results support the usefulness of stereo information, but the single-view ablation remains much stronger than several monocular baselines. This suggests that in-domain training and the added prediction architecture may explain a substantial portion of the overall improvement. The paper should more clearly separate gains from in-domain supervision, model capacity, stereo information, camera conditioning, and the specific fusion design. A matched monocular model with comparable trainable capacity and supervision would be especially informative.
  3. The evidence for physically plausible interaction is incomplete. The framework optimizes hand and object trajectories independently and does not impose explicit contact or penetration constraints. Accurate metric placement can reduce hovering and interpenetration, but it does not guarantee physically valid interaction. The reported geometric and trajectory metrics are useful, yet they do not directly quantify penetration or contact consistency. Since improved physical alignment is a central motivation, I would like to see direct measurements of hand–object penetration and contact quality, together with failure cases under severe occlusion. The claims should distinguish improved metric alignment from demonstrated physical plausibility.
  4. The related-work discussion could be broadened. The authors are encouraged to cite and discuss “From 2D Alignment to 3D Plausibility: Unifying Heterogeneous 2D Priors and Penetration-Free Diffusion for Occlusion-Robust Two-Hand Reconstruction,” which explores foundation-model priors and physical plausibility in hand reconstruction. “SignAvatar: Sign Language 3D Motion Reconstruction and Generation” is also relevant to the broader context of 3D motion reconstruction, although its focus differs from stereo hand-object interaction. Briefly discussing these works would help readers understand the connections and differences between StereoHOI and related approaches.

I would be happy to read the authors’ rebuttal and reconsider my score.

Questions for authors
  1. How does the proposed alternating cross-view transformer compare with a simpler fusion network using identical frozen features, supervision, and downstream processing?

  2. Is the stereo transformer initialized randomly or from pretrained weights? What are the trainable parameter count and complete training configuration?

  3. How much of the improvement over external monocular baselines comes from in-domain training, and how much comes from calibrated stereo input?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.