- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes EgoHSI3R, an online, feed-forward model that jointly recovers camera motion, dense scene geometry, world-space bilateral MANO hands and per-vertex hand contact from monocular egocentric video. It builds on the recurrent reconstruction model CUT3R and the hand estimator WiLoR, which are connected by two components. The Hand Prompt Builder fuses WiLoR features of each hand crop with local scene features and injects the resulting left/right hand tokens into CUT3R's decoder. The Interaction-Aware Contact Builder maps the decoded hand tokens to MANO vertices, predicts per-vertex contact from local scene features, and feeds this information back to refine the hand estimate. The model is trained on six egocentric hand-object datasets, with contact labels derived from 1 cm hand-object proximity.
The experiments cover camera-space hand reconstruction on ARCTIC and HOT3D, world-space hand reconstruction on ARCTIC, contact estimation on ARCTIC and DexYCB, and depth and camera estimation on H2O, together with an ablation of WiLoR fine-tuning and contact feedback. The headline result is a world-space W-MPJPE of 108.80 mm on ARCTIC, compared with 124.59 mm for the strongest baseline, StableHand, at about 10 FPS.
Critical strengths
A complete, causal output that existing world-space hand methods do not provide. The world-space baselines in Table 2 (HaWoR, StableHand, Dyn-HaMR, HMP-SLAM), as well as HandFlow and MINT, all rely on offline components such as SLAM with bundle adjustment, sequence-level optimization, infilling or bidirectional windows. EgoHSI3R uses only past frames, yet at about 10 FPS and 6.7 GiB it still produces camera poses, scene pointmaps, bilateral MANO hands and dense contact in a single pass (Table 8). The matched-only accuracy of the detected hands is good (p-free PA of 7.0/6.3 mm on ARCTIC/HOT3D), and H2O depth improves over the online TTT3R baseline (Abs Rel from 0.045 to 0.027). This matters to my assessment because a single causal model producing all of these quantities is practically useful for streaming egocentric applications, and it removes the per-sequence alignment that modular pipelines need.
Contact is used as a feedback signal, with a controlled ablation. The Contact Builder maps joint tokens to vertices through the fixed MANO skinning weights, which adds no learned topology, and its feedback path is initialized at zero, so training starts from the pretrained reconstruction solution. Table 5 compares variants with and without feedback under the same warm-up and joint-training schedule. With feedback, MPJPE improves on both ARCTIC and HOT3D, ARCTIC W-MPJPE improves from 113.3 to 108.8 mm, and contact F1 also increases. This matters because it is the most direct evidence in the paper that coupling hand reconstruction with interaction cues helps, rather than the two merely co-existing in one network. More broadly, extending recurrent feed-forward 4D reconstruction from bodies to articulated hands and contact is a timely and natural direction for egocentric and embodied-learning applications.
Critical weaknesses
The positioning and the main quantitative comparisons are not yet sufficiently supported. The three issues below concern the paper's main claims, which is why they dominate my assessment.
1, the positioning relative to prior work is unclear. Hand3R (arXiv ID: 2602.03200) addresses the same task with a very similar design: it fuses hand-expert crop features with box-pooled CUT3R scene features into a prompt that is injected into CUT3R's recurrent state, and it describes itself as "the first online framework for joint 4D hand-scene reconstruction". It is neither cited nor compared. In addition, Human3R is described as offline, although it is an online, CUT3R-based method. Relative to these works, the novelty appears to lie mainly in the bilateral gated prompts and the contact branch.
2, the comparability of the headline results is unclear. As far as I can tell, the baseline numbers are taken verbatim from prior papers without this being stated: those in Table 2 from StableHand (arXiv 2605.18553), and those in Table 1 from ACE-Ego-Hand. StableHand's numbers were obtained on the held-out ARCTIC subject s05, with 150-frame clips and first-frame similarity alignment. EgoHSI3R, in contrast, is evaluated on a self-defined recording-level split whose protocol is not described. ARCTIC W-MPJPE is highly protocol-sensitive: HaWoR scores 171.6 mm in StableHand but about 95 mm under another recent protocol. The 15.8 mm headline improvement may therefore reflect differences in test data and protocol rather than differences between the methods. In Table 1, the "best PA-p" claim relies on matched-only (p-free) numbers that are given only for the proposed method. Moreover, the method's penalized PA-p implies a miss rate (about 0.2%) more than ten times lower than the one implied by its MPJPE-p (about 3%), which suggests that missed hands are charged a smaller penalty than for the baselines.
3, the joint hand-scene claim is not directly evaluated. The work is motivated by floating hands and hand-scene interpenetration, yet hands, scene and contact are each evaluated only against their own ground truth, and no metric relates the predicted hands to the predicted scene. It is also unclear whether hands and scene actually benefit from each other. No ablation removes the scene context from the hand prompts. Table 4 omits vanilla CUT3R (the initialization) and a same-data control without the hand branch. The depth gain over TTT3R, the only online baseline, compares a model trained with supervision on H2O against a zero-shot one.
Questions for authors
-
Could you discuss Hand3R (arXiv 2602.03200) and clarify which components of EgoHSI3R are new relative to it? A quantitative comparison, at least on overlapping datasets such as DexYCB or HOI4D, would make the contribution much clearer.
-
Were the baseline numbers in Tables 1 and 2 re-run on your split or taken from StableHand and ACE-Ego-Hand? If they were taken from those papers, could you evaluate EgoHSI3R under exactly their protocols, namely StableHand's s05 split (155 clips of 150 frames, first-frame similarity alignment) and ViDiHand/ACE's fixed ARCTIC and HOT3D segments? Alternatively, could you re-run the baselines on your split? World-space results on HOT3D, the standard benchmark for this task, would further strengthen the evaluation. This is the most important point for my assessment.
-
Which placeholder is used for missed hands in your PA-p, and is it the same as for the baselines? Could you report detection recall, as in the source tables, together with matched-only numbers for the baselines? Please also clarify the statement "penalized even with GT boxes" and which hand detector is used at inference.
-
Could you report (a) vanilla CUT3R, and CUT3R fine-tuned on the same data without the hand branch, in Table 4, and (b) EgoHSI3R without the scene context in the hand prompts? Which CUT3R parameters are trained, and does scene quality degrade on non-egocentric benchmarks?
-
Since floating hands and interpenetration motivate the work, could you report metrics that relate the predicted hands to the predicted scene? Examples include penetration depth into the predicted pointmap, the distance from predicted contact vertices to the nearest scene point, and agreement between predicted contact and the reconstructed geometry. Please compare these against a modular pipeline (e.g., WiLoR + CUT3R with post-hoc alignment).
-
Could you clarify the target application of the proposed setting? Data generation for embodied learning is naturally offline, where future frames and global smoothing are available, while online AR/VR devices typically have multi-camera and IMU input. In addition, the contact labels describe proximity to the annotated manipulated object rather than to the scene. Which concrete application requires causal monocular inference, and how does the method compare there to an offline variant of itself?