acceptodds
Under review as a conference paper at ICLR 2027

EgoH4RT: Bringing Egocentric Hands and Camera 4D Reconstruction Together

18 views

Abstract

Reconstructing hands from egocentric video requires preserving fine hand articulation while recovering camera motion in a consistent world coordinate system. This becomes more difficult over long videos, where errors in local estimates can accumulate. To address these challenges, we introduce EgoH4RT, a framework that combines the articulation priors of a pretrained hand model with the geometric understanding of a 3D foundation model. Hand queries connect these complementary representations, drawing on evidence across frames to recover detailed bimanual poses alongside metric camera motion. To carry these local predictions into a coherent reconstruction of the full video, we jointly optimize camera constraints spanning short and long time intervals, then refine world-space hand motion within the camera path. The model learns from approximately 2,600 hours of video across 14 egocentric sources through a training recipe that accommodates different joint definitions and incomplete annotations. Experiments show that EgoH4RT outperforms multi-stage reconstruction pipelines in world-space hand and camera reconstruction and methods built on pretrained video diffusion models in camera-space hand reconstruction.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
egocentric vision, camera pose estimation, 3D hand reconstruction, long-video reconstruction

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_z2PxOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes EgoH4RT, a feed-forward model that jointly estimates bimanual MANO parameters and a metric camera trajectory from egocentric RGB video. Within each 16-frame window, hand queries built from WiLoR features attend to Pi3X video-geometry features to decode both hands, while Pi3X predicts the metric camera poses. For long videos, windows sampled at several frame strides provide relative camera-pose constraints. These are fused by pose graph optimization (PGO), and the MANO parameters are then refined with the camera trajectory fixed. The model is trained on about 2,600 hours of video from 14 egocentric datasets with heterogeneous annotations. On HOT3D, OakInk2, and Xperience, it outperforms or matches baselines such as HaWoR, MINT, and HandFlow. This holds for camera-space hand reconstruction, short-window camera estimation, and long-horizon world-space reconstruction.

Critical strengths
  1. Clear goal and unified design. The paper estimates bimanual MANO parameters and a metric camera trajectory with a single feed-forward model, instead of running a separate hand estimator and a SLAM system. This avoids the error accumulation of multi-stage pipelines. The problem also matters for learning from egocentric human video.
  2. Strong empirical results, especially for long videos. On long-horizon HOT3D sequences (Table 4), world-space hand error (W-MPJPE) drops to 44.5 mm, compared with 108.1 mm for HaWoR and 61.1 mm for HandFlow. Camera ATE drops to 5.3 mm, compared with 21.2 mm for HandFlow. The method also outperforms or matches the baselines in camera-space hand reconstruction on HOT3D and OakInk2 (Table 2), and in short-window camera estimation against VGGT- and Pi3-based models (Table 3).
Critical weaknesses
  1. Limited technical novelty The short-window model mainly combines two pretrained models, Pi3X and WiLoR. The long-video aggregation uses standard PGO. PGO is common in visual SLAM and has already been used to build long-sequence reconstructions from feed-forward geometry models, e.g., MonST3R (Zhang et al., 2025b) and VGGT-Long (Deng et al., 2025; not cited).

  2. Unclear positioning against the closet work MINT (Zhu et al., 2026) and ReViV (Lyu et al., 2026) also estimate camera and hand motion jointly, but the paper does not clearly explain how its short-window pipeline differs from them. The relevant parts of the Related Work section (L119–123 and L127–131) are hard to follow. As a result, it is hard to identify the paper’s own technical contribution.

  3. Insufficient experimental validation

  • The PGO and hand refinement could likely also be applied to existing short-window methods such as MINT and ReViV. Without that comparison in Sec. 4.2.3, it is unclear whether the large gains in Table 4 come from the proposed window model or from the aggregation step.
  • The ablation in Table 7 shows only modest gains from these components.
  • Most of the evaluation, including all world-space hand results, uses HOT3D only. Other common benchmarks such as ARCTIC are not used.
Questions for authors
  1. How does the short-window pipeline differ from MINT and ReViV in architecture, supervision, and training data? Which of these differences explain the improvements? (clarification)
  2. Can the PGO and hand refinement be applied to the outputs of MINT or ReViV? If so, how do these combinations compare with EgoH4RT in the setting of Sec. 4.2.3? (could change my assessment)
  3. What is new in the multi-stride PGO compared with existing global optimization over pairwise or chunk predictions, such as MonST3R and VGGT-Long? (clarification)
  4. Can the authors report results on another benchmark, e.g., a held-out split of ARCTIC, especially for world-space hand reconstruction? (could change my assessment)
  5. In Eq. (3), the shape coefficients β are indexed by frame n. Shouldn’t hand shape be constant over a sequence? Appendix B.3 already shares a shape offset within each run. (clarification)
  6. Please cite the baseline methods again in the Experiments section. This would make the tables easier to follow. (suggestion)
Reviewer_o3RjOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

EgoH4RT combines pretrained hand features with video geometry to reconstruct hand motion and metric camera. The model was trained with mixed source of data and shows improvement over previous method on accuracy and long-horizon handling.

Critical strengths

Strength:

  • Strong results: The method shows substantial improvements over previous methods, espcially on camera estimation.
  • Technical soundness: The motivation of using pretrained Pi3X model to estimate camera+hand poses makes sense to me.
  • Generalisation: The zero-shot results on OakInk2 shows the models has potential of a good generalization ability, although I would encourage the authors to show more in-the-wild results
  • Overcoming detection gaps: The proposed method support accurate estimation even when the hands are failed to be detected, which is a desired by real-world usage. The quantitative results support this advantage.
Critical weaknesses

Weakness:

  • Long horizon ability on hand estimation: The paper's HOT3D’s detection-filtered hand segments average approximately 142 frames and evaluate only right hands, why? longer Xperience videos report only camera results, why?
  • Architectural design choices: I would like to see some results on "injecting the new queries as extra tokens and feed them to the pretrained Pi3X model" vs the current method that introduces more learnable parameters from scratch.
  • Camera estimation results in different root-moving patterns: I would like to see more results on how accurate the camera estimation is in different motion patterns, e.g., estimating the camera movement when the person is working or running is definatelly harder than a person standing there doing handy works.
  • Point-track comparison: Consider evaluating the proposed method with point-tracking based methods; for example, CoTracker3–MANO.
  • Data recipe: the method use a full bunch of different training sources, I would like to see a detailed ablation on which dataset helps how much, and perhaps a scaling law of the training scale.
  • Please provide a detailed formulation and pipeline of "LONG-VIDEO POSE GRAPH OPTIMIZATION AND HAND REFINEMENT", I recommend the authors to provide pseudo code for this part of the method.
  • Extending to causal prediction: I would like to see the result when the input video comes as a stream and how this model can handel causal predictions.
Questions for authors

Please see Weakness, I have put my questions there.

1 comment
Reviewer_o3RjOpenReview ↗

Score updating after rebuttal

The author rebuttal addressed all my concerns so I am happy to keep my rating as weak accept.

Reviewer_ecXDOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Authors introduced EgoH4RT, a framework that is designed to estimate bimanual hand parameters and camera motions together from egocentric videos. It combines a pretrained 3D foundation model, Pi3X with WiLoR hand pose estimator using hand-specific queries and attention mechanisms. For long videos, it adopts pose graph optimization on multiple results obtained from different temporal locations. Experiments showed the improved hand pose estimation and camera estimation accuracies, while keeping the memory consumption nearly constant.

Critical strengths
  • Well-motivated problem and thoughtful system design: The proposed integration of hand-specific queries with video-geometry representations provides a more unified approach than conventional pipelines that combine separate hand estimation and SLAM components.

  • Computational efficiency: The method maintains nearly constant GPU memory consumption across sequences ranging from 128 to 1,802 frames.

Critical weaknesses
  • Missing baseline: One of the important claim of the paper is that jointly estimating hand poses and camera trajectories is more effective than combining separate estimation pipelines. However, a direct comparison using the same backbone components in a two-stage pipeline is missing. Without this baseline, it is difficult to determine whether the improvements arise from the proposed cross-attention fusion or from the stronger camera backbone and extensive training data.

  • Unclear contribution of training data versus methodology: EgoH4RT is trained on approximately 2,600 hours of egocentric video collected from 14 sources, substantially more than the training data used by competing approaches such as HaWoR and HandFlow. Consequently, it is difficult to distinguish the benefits of the proposed architecture from those of increased training data and diversity.

  • Potentially unfair comparison in Table 3: The comparison with VGGT-Ω, DA3, StreamVGGT, and vanilla Pi3X does not fully account for differences in training conditions. While these methods are evaluated in a zero-shot setting, EgoH4RT benefits from egocentric fine-tuning and explicit camera supervision. Therefore, the reported improvement in absolute trajectory error (ATE) from 3.98 mm to 1.34 mm over Pi3X may largely reflect domain adaptation rather than the proposed joint estimation framework. Including a camera-only Pi3X baseline fine-tuned under comparable conditions would help isolate the contribution of the proposed architecture.

  • Insufficient positioning within existing literature: The related work section does not adequately establish how EgoH4RT differs from existing approaches or clearly identify the limitations that motivate its architectural choices. A more comprehensive discussion of closely related methods, particularly those addressing joint hand-camera estimation and egocentric reconstruction, would help clarify the novelty of the proposed framework and better contextualize its contributions.

Questions for authors
  • The competing methods generally rely on weaker camera estimators and substantially less training data, making it difficult to attribute the reported improvements to the proposed architecture. How do you think for this aspect?

  • The ablation results suggest that hand articulation accuracy largely comes from the frozen WiLoR backbone, as the PA-MPJPE on OakInk2 does not improve over WiLoR alone. A crucial baseline combining WiLoR hand predictions with Pi3X camera estimates and the same pose graph optimization is missing, leaving the effectiveness of the query-based fusion unclear. Please share your thoughts on this.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.