acceptodds
Withdrawn from ICLR 2027

EgoHSI3R: Online 4D Hand-Scene Interaction Reconstruction from Egocentric Video

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Egocentric videos offer an abundant source of hand-scene interactions for embodied learning, which requires recovering hands, camera, and scene consistently in a shared 3D world. Existing methods focus on either camera-frame hands or world-space hand trajectories, without jointly recovering the scene. Building a complete reconstruction from such specialized predictions therefore typically requires per-sequence optimization. We propose EgoHSI3R, a unified feed-forward framework for online 4D hand-scene interaction reconstruction from egocentric monocular video. To jointly recover these components, it maintains a recurrent scene representation that accumulates geometric evidence over time and provides a shared world space and scale. We introduce hand-centric prompts that fuse pretrained hand priors and local scene context, while MANO vertex queries attend to scene features to predict dense per-vertex contact. These designs enable a single network to jointly estimate camera motion, dense scene geometry, world-space bilateral MANO hands, and per-vertex hand contact. EgoHSI3R achieves the lowest world-space hand joint error on ARCTIC, while running faster than prior world-space hand pipelines and retaining competitive camera and scene reconstruction quality.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D Vision, Hand-Scene Interaction, 4D Reconstruction, Egocentric Vision

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_iAVnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes EgoHSI3R, an online, feed-forward model that jointly recovers camera motion, dense scene geometry, world-space bilateral MANO hands and per-vertex hand contact from monocular egocentric video. It builds on the recurrent reconstruction model CUT3R and the hand estimator WiLoR, which are connected by two components. The Hand Prompt Builder fuses WiLoR features of each hand crop with local scene features and injects the resulting left/right hand tokens into CUT3R's decoder. The Interaction-Aware Contact Builder maps the decoded hand tokens to MANO vertices, predicts per-vertex contact from local scene features, and feeds this information back to refine the hand estimate. The model is trained on six egocentric hand-object datasets, with contact labels derived from 1 cm hand-object proximity.

The experiments cover camera-space hand reconstruction on ARCTIC and HOT3D, world-space hand reconstruction on ARCTIC, contact estimation on ARCTIC and DexYCB, and depth and camera estimation on H2O, together with an ablation of WiLoR fine-tuning and contact feedback. The headline result is a world-space W-MPJPE of 108.80 mm on ARCTIC, compared with 124.59 mm for the strongest baseline, StableHand, at about 10 FPS.

Critical strengths

A complete, causal output that existing world-space hand methods do not provide. The world-space baselines in Table 2 (HaWoR, StableHand, Dyn-HaMR, HMP-SLAM), as well as HandFlow and MINT, all rely on offline components such as SLAM with bundle adjustment, sequence-level optimization, infilling or bidirectional windows. EgoHSI3R uses only past frames, yet at about 10 FPS and 6.7 GiB it still produces camera poses, scene pointmaps, bilateral MANO hands and dense contact in a single pass (Table 8). The matched-only accuracy of the detected hands is good (p-free PA of 7.0/6.3 mm on ARCTIC/HOT3D), and H2O depth improves over the online TTT3R baseline (Abs Rel from 0.045 to 0.027). This matters to my assessment because a single causal model producing all of these quantities is practically useful for streaming egocentric applications, and it removes the per-sequence alignment that modular pipelines need.

Contact is used as a feedback signal, with a controlled ablation. The Contact Builder maps joint tokens to vertices through the fixed MANO skinning weights, which adds no learned topology, and its feedback path is initialized at zero, so training starts from the pretrained reconstruction solution. Table 5 compares variants with and without feedback under the same warm-up and joint-training schedule. With feedback, MPJPE improves on both ARCTIC and HOT3D, ARCTIC W-MPJPE improves from 113.3 to 108.8 mm, and contact F1 also increases. This matters because it is the most direct evidence in the paper that coupling hand reconstruction with interaction cues helps, rather than the two merely co-existing in one network. More broadly, extending recurrent feed-forward 4D reconstruction from bodies to articulated hands and contact is a timely and natural direction for egocentric and embodied-learning applications.

Critical weaknesses

The positioning and the main quantitative comparisons are not yet sufficiently supported. The three issues below concern the paper's main claims, which is why they dominate my assessment.

1, the positioning relative to prior work is unclear. Hand3R (arXiv ID: 2602.03200) addresses the same task with a very similar design: it fuses hand-expert crop features with box-pooled CUT3R scene features into a prompt that is injected into CUT3R's recurrent state, and it describes itself as "the first online framework for joint 4D hand-scene reconstruction". It is neither cited nor compared. In addition, Human3R is described as offline, although it is an online, CUT3R-based method. Relative to these works, the novelty appears to lie mainly in the bilateral gated prompts and the contact branch.

2, the comparability of the headline results is unclear. As far as I can tell, the baseline numbers are taken verbatim from prior papers without this being stated: those in Table 2 from StableHand (arXiv 2605.18553), and those in Table 1 from ACE-Ego-Hand. StableHand's numbers were obtained on the held-out ARCTIC subject s05, with 150-frame clips and first-frame similarity alignment. EgoHSI3R, in contrast, is evaluated on a self-defined recording-level split whose protocol is not described. ARCTIC W-MPJPE is highly protocol-sensitive: HaWoR scores 171.6 mm in StableHand but about 95 mm under another recent protocol. The 15.8 mm headline improvement may therefore reflect differences in test data and protocol rather than differences between the methods. In Table 1, the "best PA-p" claim relies on matched-only (p-free) numbers that are given only for the proposed method. Moreover, the method's penalized PA-p implies a miss rate (about 0.2%) more than ten times lower than the one implied by its MPJPE-p (about 3%), which suggests that missed hands are charged a smaller penalty than for the baselines.

3, the joint hand-scene claim is not directly evaluated. The work is motivated by floating hands and hand-scene interpenetration, yet hands, scene and contact are each evaluated only against their own ground truth, and no metric relates the predicted hands to the predicted scene. It is also unclear whether hands and scene actually benefit from each other. No ablation removes the scene context from the hand prompts. Table 4 omits vanilla CUT3R (the initialization) and a same-data control without the hand branch. The depth gain over TTT3R, the only online baseline, compares a model trained with supervision on H2O against a zero-shot one.

Questions for authors
  1. Could you discuss Hand3R (arXiv 2602.03200) and clarify which components of EgoHSI3R are new relative to it? A quantitative comparison, at least on overlapping datasets such as DexYCB or HOI4D, would make the contribution much clearer.

  2. Were the baseline numbers in Tables 1 and 2 re-run on your split or taken from StableHand and ACE-Ego-Hand? If they were taken from those papers, could you evaluate EgoHSI3R under exactly their protocols, namely StableHand's s05 split (155 clips of 150 frames, first-frame similarity alignment) and ViDiHand/ACE's fixed ARCTIC and HOT3D segments? Alternatively, could you re-run the baselines on your split? World-space results on HOT3D, the standard benchmark for this task, would further strengthen the evaluation. This is the most important point for my assessment.

  3. Which placeholder is used for missed hands in your PA-p, and is it the same as for the baselines? Could you report detection recall, as in the source tables, together with matched-only numbers for the baselines? Please also clarify the statement "penalized even with GT boxes" and which hand detector is used at inference.

  4. Could you report (a) vanilla CUT3R, and CUT3R fine-tuned on the same data without the hand branch, in Table 4, and (b) EgoHSI3R without the scene context in the hand prompts? Which CUT3R parameters are trained, and does scene quality degrade on non-egocentric benchmarks?

  5. Since floating hands and interpenetration motivate the work, could you report metrics that relate the predicted hands to the predicted scene? Examples include penetration depth into the predicted pointmap, the distance from predicted contact vertices to the nearest scene point, and agreement between predicted contact and the reconstructed geometry. Please compare these against a modular pipeline (e.g., WiLoR + CUT3R with post-hoc alignment).

  6. Could you clarify the target application of the proposed setting? Data generation for embodied learning is naturally offline, where future frames and global smoothing are available, while online AR/VR devices typically have multi-camera and IMU input. In addition, the contact labels describe proximity to the annotated manipulated object rather than to the scene. Which concrete application requires causal monocular inference, and how does the method compare there to an offline variant of itself?

Reviewer_ZzQJOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

Authors proposed EgoHSI3R, an online feed-forward model that jointly recovers camera motion, scene geometry, world-space MANO hands, and dense hand contact from egocentric video. It builds on CUT3R with WiLoR hand priors and a vertex-level contact head. It achieves the best world-space hand error on ARCTIC with faster runtime than prior pipelines.

Critical strengths
  • Unified framework: The paper addresses several related tasks in one framework by jointly estimating camera motion, scene geometry, bilateral hands, and dense contact in a shared 3D space. The setting is also practical by only using current and previous frames.

  • Efficiency: It runs at 10.07 FPS with 6.65G memory usage. It is faster and lighter than HaWoR and HandFlow.

Critical weaknesses
  • Limited novelty: The method is largely seen as the combination of CUT3R and WiLoR with Human3R-style visual prompting adapted to hands. This may not be enough for top-tier conference idea. Also, each component such as gated RoIAlign fusion, vertex-query cross-attention is also not technically new.

  • Core claim seems not properly quantified: The paper is motivated to solve floating hands and hand-scene interpenetration challenges, but seemingly no metric is measuring such aspects. Further, contact is only evaluated as vertex labels, and its GT is hand-object contact, not hand-scene contact.

  • Narrow evaluation: World-space hands are evaluated only on ARCTIC and scenes are evaluated only on H2O, both from the training domain. There are also no in-the-wild results (e.g., Ego4D) or long-sequence analysis.

  • Missing ablations: There is no ablation for the scene conditioning in the hand prompts and no comparison with vanilla CUT3R.

Questions for authors
  • The method performs well in world-space position accuracy, but acceleration error and camera-space jitter are higher than some temporal hand-reconstruction baselines. I would like to ask authors how they interpret the results.
Reviewer_hiyQOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

An online feed-forward model that jointly predicts camera poses, dense scene pointmaps, bilateral world-space MANO hands, and per-vertex contact from monocular egocentric video. It builds on CUT3R and WiLoR, adding scene-conditioned hand prompts and a contact head whose gated feedback refines the hands. Validation covers world-space hands on ARCTIC (W-MPJPE 108.80 vs 124.59 mm for StableHand), camera-space hands, contact vs HACO, scene and camera on H2O, an ablation, and runtime (10.1 FPS).

Critical strengths
  • Best world-space accuracy in a fully online setting. It has the lowest W-MPJPE and WA-MPJPE on ARCTIC while staying causal and running faster than HaWoR and HandFlow, which is the setting egocentric embodied learning needs.

  • Contact feedback is validated. A controlled ablation shows feedback lowers W-MPJPE from 113.30 to 108.80 mm and raises contact F1, supporting the claim that interaction cues improve reconstruction.

Critical weaknesses
  • The core consistency claim is never measured. There is no penetration or floating metric and no hand+scene composition baseline, so the main motivation (avoiding floating and interpenetration) is unsupported.
  • Limited novelty and attribution.
    • The CUT3R + visual-prompt recipe follows Human3R, which the paper calls offline although it is online.
    • The scene gating is not ablated, and the largest gain comes from WiLoR fine-tuning.
    • Outside W-MPJPE, results trail baselines: camera-space wins only under p-free, jitter is high, and contact F1 is below HACO.
Questions for authors
  • Can you report penetration and floating metrics against a hand+scene baseline such as your WiLoR+ViPE pipeline?
  • How do baselines score under p-free, and what fraction of GT-visible hands is missed?
  • How does accuracy change without the scene-context gate?
  • What is new relative to Human3R beyond the hand and contact heads?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.