acceptodds
Under review as a conference paper at ICLR 2027

EgoFound3R: End-to-End Egocentric Hand Reconstruction in World Space with Point-Wise Interaction Attributes

23 views

Abstract

Egocentric video has become a primary source of supervision for embodied models, and its value rests on recovering hand motion in world coordinates, which camera motion and hand occlusion make difficult. Existing reconstruction pipelines typically separate hand and scene estimation, leave interaction attributes to separate task-specific models, and invoke several models per video, so no prior reconstruction model estimates these attributes and throughput becomes a practical constraint on large-scale annotation. We therefore introduce EgoFound3R, a unified end-to-end model that estimates world-space hand geometry in a metric scale shared with the scene, and predicts point-wise interaction attributes, including visibility, contact, and distance. The model integrates three designs: (i) structured hand prompts that transfer pretrained geometric priors to world-space hand reconstruction; (ii) an explicit hand representation that decodes hand geometry and interaction attributes; and (iii) a shared-parameter multi-rate design that lowers inference cost. Together, these designs predict hand geometry and point-wise attributes in one pass. On OakInk-v2, TACO, and HOI4D, EgoFound3R reduces the mean per-joint position error (MPJPE) by 43.2%, 22.4%, and 11.6% over previous methods and predicts point-wise contact and distance alongside the geometry in the same pass, while attaining approximately 6× higher throughput.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
World Space, Hand Reconstruction, Contact Estimation, Egocentric Vision

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ZRbaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

EgoFound3R combines a frozen geometry backbone, structured hand prompts, and multirate aggregation to predict world-space hands and point-wise visibility, contact, and distance from monocular egocentric video. Experiments report improved raw joint accuracy on three benchmarks and approximately sixfold throughput over WiLoR (Secs. 3–4; Tables 1–4 and 6).

Critical strengths
  1. Jointly decoding geometry and interaction attributes without an external hand detector or object template is a useful contribution for egocentric annotation, and the model improves both raw joint error and joint-level contact F1 on the three common evaluation sets (Secs. 3.2 and 3.4; Tables 1–2).
  2. The ablations substantiate the architectural choices: on TACO, replacing explicit points with MANO regression or removing hand-to-scene feedback increases raw joint error from 54.64 mm to 86.66 or 78.82 mm, respectively (Table 4). The shared-parameter stride study also directly examines the accuracy–efficiency trade-off (Appendices C and I, Fig. 10 and Tables 17–19).
Critical weaknesses
  1. The evidence for consistent world-space trajectories is weaker than the local hand-reconstruction results. World-space hands still compose camera-space predictions with a predicted camera trajectory, while scale refinement and overlapping-clip alignment remain post-processing (Appendix A, p. 15, lines 780–797). Sharing one scale within a clip does not by itself prevent camera-pose drift across clips. Yet all evaluation windows last 2 s, and the duration argument in Sec. 4.2, lines 318–323, is inferred from differences between datasets rather than sequence lengths. W and WA additionally fit a per-window Sim(3) using the first two or all valid frames, respectively (Appendix D, p. 18, lines 932–939 and 962–970), so these metrics do not independently validate absolute metric scale. On matched HOI4D windows, EgoFound3R's W error is worse than Dyn-HaMR's (205.48 versus 173.62 mm; Table 15). Table 10's temporal improvements and Table 20's TACO depth evaluation at the model's own scale are meaningful counterevidence, but neither establishes stability of stitched long videos. The claims should distinguish demonstrated short-window accuracy from long-video drift resistance; duration-controlled stitching evaluation would strengthen the latter in future work.
  2. The interaction results expose a material gap between accurate native-point predictions and dependable dense surface annotations. On TACO, contact F1 falls from 0.661 at 195 vertices to 0.320 at 778 vertices, with recall falling from 0.719 to 0.265 (Table 11); the matched dense comparison also favors InteractVLM, 0.458 versus 0.388 (Table 16). Moreover, distance MAE on ground-truth-contact native vertices is 31.38 mm (Table 19), exceeding the 14 mm band defining those contacts (Appendix G, p. 20, lines 1074–1079). This raises a substantive concern about using the outputs as contact/proximity supervision, beyond the missing explanation of attribute upsampling. The heads share features, but the described classification and distance objectives do not explain how their outputs are constrained to agree (Appendix B, Eqs. (4)–(8)); these aggregate errors do not themselves prove point-wise inconsistency. I acknowledge that Tables 13–14 preserve the native-point advantage under threshold and contact-definition changes, and Tables 17–19 support the architecture. Those results do not resolve dense-output degradation or establish precise proximity estimates. The paper should explain the usable resolution and consistency of its returned attributes and qualify its annotation claims accordingly.
Questions for authors
  1. [Clarification; Weakness 1] Does any existing result evaluate overlapping-clip stitching beyond the 2 s protocol? Please identify it and explain which evidence separates sequence duration from dataset difficulty in lines 318–323. If none does, can the drift claim be limited to the evaluated setting? A new long-video benchmark is not requested for rebuttal.
  2. [Clarification; Weakness 1] Please give the existing root-recovery, hand–depth scale-fusion, and clip-stitching rules, including the fallback when visible-hand support is weak (Appendix A, lines 774–797). Are Tables 1, 15, and 20 generated after the same scale-refinement procedure, and what does their combination establish about scale accuracy versus camera-trajectory error?
  3. [Clarification; Weakness 2] How are contact scores and ground-truth labels transferred from 195 to 778 vertices, and which threshold is used at each resolution? Can the existing implementation or cached evaluations explain the large recall loss and whether it arises at native vertices or newly interpolated vertices? Tables 13–14 currently address only the native point set.
  4. [Clarification; Weakness 2] What do existing outputs show about agreement between predicted contact and predicted distance, especially given TACO's 31.38 mm MAE on ground-truth contacts? Please clarify the evaluated distance quantity, any consistency mechanism beyond the losses shown, and the actual loss weights and stride-sampling proportions used for these results and ablations. Existing configurations or cached diagnostics would suffice; retraining is not requested.
3 comments
Reviewer_ZRbaOpenReview ↗

Remaining concern about metric world-space validation

Thank you for the additional evidence and detailed clarifications. The corrected contact evaluation, direct distance-head results, and consistency statistics substantially address my concerns about the interaction attributes. I no longer consider the previously reported dense-contact degradation and distance errors valid grounds for those criticisms.

My remaining concern is the validation of the final world-space trajectories. I appreciate the clarification that the drift discussion refers to the evaluated two-second windows. However, W and WA both fit a similarity transform within each window, and the camera ATE in Appendix J also uses per-window Sim(3) alignment. These results support short-window reconstruction accuracy, but leave the metric accuracy of the composed world-space trajectories unclear because window-level scale errors can be removed by the alignment.

The raw hand errors and depth results at the model's own scale provide useful evidence. Still, they evaluate camera-space hand geometry and scene depth separately. The final world-space trajectory also depends on the predicted camera motion and, for longer videos, the alignment between clips. The qualitative demonstrations and explanation of the stitching procedure do not yet quantify the accumulated error in that final output.

This is a narrower concern than in my initial review, and I appreciate the substantial improvements in the response. However, it remains important to my assessment of the world-space contribution, so I will maintain my original recommendation.

Thank you for your follow-up. We appreciate your recognition of the corrected interaction-attribute results and your clarification of the remaining concern regarding the world-space trajectories.

To examine the effect of scale alignment, we recomputed W and WA using rigid SE(3) alignment. SE(3) fits rotation and translation while fixing the scale to 1. The Sim(3) results below correspond to the corresponding rows of the revised Table 1. For each method, the Sim(3) and SE(3) results use the same valid samples.

DatasetMethodW Sim(3)W SE(3)WA Sim(3)WA SE(3)
OakInk-v2HaWoR100.46131.7237.4778.51
ReViV80.6682.4929.4549.28
EgoFound3R30.8831.1918.7720.23
TACOHaWoR86.3899.5135.3459.86
ReViV55.4056.4030.3733.68
EgoFound3R40.0340.9725.1627.38
HOI4DHaWoR171.09197.6255.9681.22
ReViV260.61292.2962.37136.12
EgoFound3R159.70167.3249.8461.15

All errors are in mm, and lower is better.

We further evaluated stitched 120-frame long trajectories, each formed by concatenating adjacent 60-frame windows from the same sequence. The three methods are compared on their common valid hand frames.

DatasetMethodW Sim(3)W SE(3)WA Sim(3)WA SE(3)
OakInk-v2HaWoR91.8797.9937.9366.86
ReViV175.89163.7443.7092.86
EgoFound3R47.4147.6323.7728.80
TACOHaWoR171.47199.0743.97109.52
ReViV74.3575.4238.8842.60
EgoFound3R44.7445.0228.3730.70
HOI4DHaWoR235.07262.9262.0296.18
ReViV376.93414.4969.69183.14
EgoFound3R215.12227.2954.8668.93

The SE(3) columns in both tables fix the scale while still fitting rotation and translation, so they can be compared directly with the Sim(3) results on the same samples. The close agreement between the Sim(3) and SE(3) errors indicates that EgoFound3R's predicted trajectories are already metrically accurate, so removing per-window scale fitting changes the W/WA errors only marginally.

We hope these additional results help clarify the metric-scale behavior of the composed world-space hand trajectories. If our responses have adequately addressed your concerns, we would be grateful if you could consider raising your score. Thank you for your time and consideration.

We would also like to provide some context for the trajectory lengths used in our evaluation. ReViV evaluates hand reconstruction on 60-frame clips, Dyn-HaMR evaluates global hand motion on 128-frame segments, and HaWoR's released evaluation code computes W/WA in chunks of up to 100 frames. Our comparison is therefore comparable in length to these established evaluation protocols.

We hope this context helps clarify the rationale for our evaluation. We also appreciate your interest in longer trajectories, and this would be a valuable direction for future work. Thank you again for your constructive feedback.

Reviewer_h5JGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes EgoFound3R, an end-to-end framework for egocentric hand reconstruction. The method jointly recovers world-space hand geometry and predicts point-wise interaction attributes, including visibility, contact, and distance. It introduces structured hand prompts, an explicit hand representation, and a multi-rate design to improve reconstruction accuracy and inference efficiency. The method is evaluated on OakInk-v2, TACO, and HOI4D, achieving strong performance in hand reconstruction, interaction prediction, and throughput compared with previous approaches.

Critical strengths
  1. The method addresses the limitations of previous multi-stage pipelines by jointly predicting hand geometry and interaction attributes in a single model. This is a meaningful and interesting direction.

  2. Besides hand reconstruction, the paper also evaluates contact prediction, visibility prediction, and inference efficiency. This makes the contribution broader than improving only one reconstruction metric.

  3. The multi-rate design considers that hand motion and scene changes have different temporal requirements. This reduces computation while maintaining performance, which is valuable for large-scale egocentric video processing.

Critical weaknesses
  1. EgoFound3R heavily relies on the frozen VGGT-Ω backbone and extends it with prompts and adapters. Although the overall performance is strong, it is still unclear how much improvement comes from the backbone itself and how much comes from the proposed hand prompts and multi-rate design. 9041_EgoFound3R_End_to_End_Ego

  2. The evaluation is mainly conducted on three existing egocentric hand-object datasets. Although these datasets contain different scenes, it remains unclear whether the method can generalize well to different camera settings, long videos, or unseen interaction types.

  3. For some baselines, such as Dyn-HaMR and InteractVLM, the authors evaluate on 100-window subsets, while EgoFound3R follows the full evaluation protocol. Although this is explained as a way to reduce computation cost, different evaluation scales may affect the comparison.

Questions for authors
  1. EgoFound3R uses the frozen VGGT-Ω as the backbone. Would the proposed hand prompts and multi-rate design still be effective if another 3D foundation model is used, or if a smaller backbone is trained from scratch?

  2. What is the trade-off between the scene update frequency and performance in the multi-rate design? More analysis of different stride settings would help understand the relationship between accuracy and efficiency.

  3. Can the predicted interaction attributes (contact, visibility, and distance) improve downstream robotic manipulation or embodied learning tasks?

  4. For baselines evaluated on different subsets, can the authors provide more comparisons under the same evaluation protocol to ensure a fair comparison?

Reviewer_bPLVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

EgoFound3R jointly reconstructs hands and scene context from egocentric video and predicts point-wise visibility, contact and distance. It adapts a frozen geometry backbone through structured hand prompts and uses different temporal rates for hand and scene processing. Evaluation covers OakInk-v2, TACO and HOI4D, with geometry, interaction and efficiency measurements.

Critical strengths
  1. Jointly predicting hand geometry and interaction attributes is useful for annotating egocentric video. The paper reports improvements in raw hand accuracy and joint-level contact prediction across three datasets.
  2. The multi-rate design addresses a practical efficiency problem. The backbone, interaction and rate ablations provide supporting evidence for the design choices.
Critical weaknesses
  • The ReViV comparison needs a convincing explanation. Table 1 reports PA-MPJPE of 44.14, 40.14 and 39.66 mm for ReViV, compared with 8.02, 8.13 and 7.99 mm for EgoFound3R. Since per-frame alignment removes global similarity differences, the gap cannot simply be attributed to camera trajectory or metric scale. It is unclear whether it reflects reconstruction quality, different training exposure, or evaluation/implementation differences. The gap alone does not establish an error, but it weakens confidence in this important baseline comparison.
Questions for authors
  • Can the ReViV setup reproduce its reported results on an overlapping benchmark before applying this paper's evaluation? If so, what explains the approximately fivefold PA-MPJPE gap here, and how does training exposure differ?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.