acceptodds
Under review as a conference paper at ICLR 2027

EgoXperience: Learning World-Grounded Egocentric Representations from Human Experience

18 views

Abstract

Reconstructing human activity from egocentric video requires more than recovering individual 3D components: the observer's motion, surrounding geometry, and hand-object interactions must fit together in a consistent world. Achieving this coherence requires both joint reconstruction and aligned supervision, yet existing methods and datasets offer limited support for either. We present EgoXperience, a full-stack paradigm that addresses these needs through data collection, automatic annotation, unified modeling, and benchmarking. Central to EgoXperience is an automatic annotation engine, EgoX-Engine, that couples geometrically verified object tracking with SLAM-based world alignment, yielding consistent supervision for the camera, scene, hands, and objects. This yields EgoX-Data-6M, a 50-hour dataset of task-driven manipulation and locomotion within and across indoor and outdoor environments, with second-level task captions. We carefully select one hour across these activity categories as EgoX-Bench to evaluate joint reconstruction. To translate this supervision into reconstruction, we introduce EgoX, trained with EgoX-Data-6M and complementary annotations from additional public datasets. Given monocular egocentric video and first-frame object masks, our model achieves joint metric reconstruction of the camera, scene, hands, and objects in a shared world coordinate frame with a single forward pass. Extensive experiments demonstrate state-of-the-art performance across scene estimation and world-space hand-object pose recovery. Against the strongest evaluated baselines, EgoX reduces camera, hand, and depth errors by 68.9%, 65.6%, and 42.9%, and improves object pose scores by 28.0%.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Motion Capture, Foundation Model, Data Pipeline

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_7jmvOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes EgoXperience, a full-stack paradigm for jointly reconstructing camera pose, scene depth, two-hand MANO meshes, and rigid-object 6-DoF poses from monocular egocentric video in a shared metric world frame. It has three parts: (1) EgoX-Engine, an automatic annotation pipeline that combines stereo depth and SLAM poses from a head-mounted rig with FoundationPose tracking of pre-scanned object meshes, verified against depth and silhouettes; (2) EgoX-Data-6M, a large dataset with many objects, tasks, and second-level captions, plus a held-out EgoX-Bench; and (3) EgoX, a DINOv3-based transformer that predicts all quantities in a single forward pass and adds a geometrically coupled loss that composes hand and object predictions with the predicted camera in world space.

On HOT3D, EgoX-Bench, HOI4D, and H2O, EgoX ranks first on hand, object, and depth metrics, and the authors report large reductions in camera, hand, and depth errors and a clear gain in object pose scores over the strongest baselines. The authors acknowledge that VGGT-Ω and DA3 Nested are better on camera pose in several benchmarks.

Critical strengths
  • Unique dataset coverage. Table 1 shows no existing dataset provides 3D hands, 6-DoF objects, meshes, depth, masks, and captions across single-area, cross-area, and outdoor settings; EgoX-Data-6M does.
  • Scalable annotation without mocap. The stereo + SLAM + FoundationPose verification loop yields mutually aligned annotations without motion-capture infrastructure.
  • Efficiency and unified output. 14.60 s for 150 frames vs. 134–167 s for multi-stage pipelines; EgoX is the only compared method producing all outputs end-to-end.
Critical weaknesses
  • Train/eval overlap and asymmetric comparison. Appendix G lists HOT3D, HOI4D, H2O, HO-Cap, H2O-3D, and DexYCB in the training mix, and three of the four main benchmarks are among them. Baselines appear to be evaluated with off-the-shelf checkpoints, with no indication of equivalent in-domain fine-tuning. This undermines the "state-of-the-art" claim.
  • Circular ground truth on EgoX-Bench. Labels come from the FoundationPose-based engine, yet FoundationPose is a baseline in Table 3. The model may be mimicking the labeler's biases, and the labels are never validated against independent mocap or manual annotation.
  • No ablation of the core contribution. The coupled supervision loss (Eq. 2) is never ablated; only ScaleHead (Table 5) and DepthHead (Table 6) are. Architectural choices (block counts, token counts) are also unablated.
  • Reproducibility deferred. Appendix G states loss weights, sampling probabilities, optimizer schedule, seeds, hardware, and training time will be reported only in the final release.
  • Proprietary hardware dependence. The pipeline relies on a specific commercial rig and its processing stack, limiting the paradigm claim.
  • Rigid, pre-scanned objects only. All 101 objects are pre-scanned in a dome; generalization to unseen instances/categories and to articulated or deformable objects is untested, while several baselines target novel objects.
Questions for authors
  1. Were the baselines (VGGT-Ω, HaWoR, MINT, ReViV, HandFlow, etc.) fine-tuned on the same HOT3D/HOI4D/H2O train splits? Can the SOTA claim be reproduced with a model trained on EgoX-Data-6M only, excluding these splits?
  2. Has the accuracy of EgoX-Bench pseudo-labels been validated against independent mocap or a manually verified subset? How do you rule out shared inductive bias with FoundationPose?
  3. What is the effect of removing the local, dense, coupled, and scale losses separately?
  4. What fraction of evaluation objects is unseen during training, and how does performance on unseen categories compare?
  5. The paper uses the term entity. What exactly does it refer to in the proposed method?
  6. Depth, scale, camera, hand, and object predictions appear to be produced separately. How consistent are these outputs when evaluated in a common coordinate system? For example, does the hand region in the depth map align with the predicted hand pose? It would be helpful to visualize and evaluate consistency across other combinations of outputs as well.
  7. More details on the training and evaluation protocols are needed. For training, please provide a table summarizing the size of each dataset, the annotations used, and how missing annotations are handled (e.g., H2O lacks object masks). For evaluation, please similarly detail the size of each dataset and the scenarios evaluated.
1 comment

Thank you to the authors for the comprehensive rebuttal and the substantial additional experiments. I appreciate the effort taken to address my concerns.

Some minor limitations remain, particularly regarding the detailed reproducibility of the training configuration and the scope of the generalization claims. Nevertheless, I believe the authors have substantially addressed the main concerns raised in my original review.

Overall, the additional experiments significantly strengthen the paper. I am therefore inclined to raise my assessment, as the proposed dataset, annotation pipeline, and unified reconstruction framework are supported by considerably stronger experimental evidence after the rebuttal.

Reviewer_VzkgOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper presents EgoXperience. It includes a dataset contribution and a model contribution. For the dataset part, it presents the EGOX-Engine as an automatic annotation engine for data creation. The authors then collect the EgoX-Data-6M in various capture settings and the data contains more comprehensive annotation types than previous work. Then, the authors present the EGOX trained by the collected data. The model achieves SoTA on several benchmarks.

Critical strengths
  • It's reasonable to consider the joint prediction for depth, camera, hand, object etc in a joint space.
  • The data collection part is well aligned with the goal of this paper. The data is collected in more various settings (indoor, outdoor) and incorporated with more comprehensive annotations compared with previous work, which can be helpful for future work.
  • The trained model achieves impressive performance on multiple benchmarks.
  • A unified feedforward model predicts all things in a single forward. Compared with combining multiple models for each indivisual task, it is more effecient and able to provide more aligned and globally consistent results.
Critical weaknesses
  • It's not clear where the performance gain is from. The authors combine multiple datasets to train the model. I was wondering if the gain is from more data or is from a better joint estimation model.
  • It seems that the ablation study is not comprehensive enough. For the metric scale prediction, it's a bit unclear how the oracle scale (G) is calculated and used. The scale head plays a crucial role for the model performance, but there is a lack of dicussion about the reasons.
  • There are multiple stages and multiple models in the EgoX-Engine pipeline. It would be better to quantatively analyze the error in the EgoX-Engine, so that the users will better know the quality of the collected data.
Questions for authors

I would like to see the authors' response to the Weakness section.

  • It's not clear where the performance gain is from. The authors combine multiple datasets to train the model. I was wondering if the gain is from more data or is from a better joint estimation model. I would like to see some discussions about this or some ablation study results that can address my concern.
  • It seems that the ablation study is not comprehensive enough. For the metric scale prediction, it's a bit unclear how the oracle scale (G) is calculated and used. The scale head plays a crucial role for the model performance, but there is a lack of dicussion about the reasons. I would like to ask the authors to present more details about the experiment settings and more discussions about the function of the scale head.
  • There are multiple stages and multiple models in the EgoX-Engine pipeline. It would be better to quantatively analyze the error in the EgoX-Engine, so that the users will better know the quality of the collected data. I would like to request more analysis about the accuracy of this automatic data engine.
Reviewer_5hdAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes EgoXperience, a "full-stack" paradigm spanning data collection, automatic annotation, modeling, and benchmarking for joint 3D reconstruction from egocentric video. Its components include: -EgoX-Engine, an automatic annotation pipeline. -EgoX-Data-6M, 50 hours (~5.4M frames) of task-driven manipulation and Locomotion. -EgoX, a feed-forward model built on DINOv3 ViT-L/16 features and a 24-block transformer, which predicts camera poses, dense depth, two-hand MANO states, rigid-object 6-DoF poses, and a metric scale factor in a single forward pass from monocular RGB plus first-frame object masks.

Validation uses clips on HOT3D, EgoX-Bench, HOI4D, and H2O. EgoX ranks best on hand, object, and depth metrics on all four benchmarks and best on camera only on HOI4D.

Critical strengths

(a) The unified world-frame formulation is the right abstraction, and the paper shows it pays off across four tasks. Routing hands, objects, depth and camera through a single shared unprojection with one learned metric scale is more than a multi-task head arrangement: it makes the tasks mutually constraining in the same coordinate system, which is precisely what per-task specialist models cannot do.

(b) The data engine is a contribution independent of the model. The specific combination — dome-scanned object meshes with AprilTag metric scale, depth-verified Foundation Pose tracking, fisheye-SLAM session frames, triangulated MANO— is a credible recipe for world-grounded egocentric ground truth at a scale that the existing egocentric HOI datasets do not reach.

Critical weaknesses

(a) EgoX-Bench is self-referential and its annotation accuracy is never validated. The benchmark is one held-out hour of the authors' own pipeline output: "ground truth" produced by FoundationPose, MANO fitting and SLAM — the same families of estimator the model is being evaluated against. No annotation- accuracy study is reported (no comparison to marker-based or manually verified poses, no error bars on the mesh-scale or MANO-fit stages), so the magnitude of the label noise is unknown.

Consequently the headline numbers on EgoX-Bench cannot be read as accuracy; they measure agreement with a particular pipeline, and a model trained on that pipeline's output is structurally advantaged.

(b) The comparison tables conflate method quality with training exposure, and the ablation study never touches the paper's central claims. The baselines are run zero-shot while EgoX trains on ~280 hours including material related to the evaluation domains; this asymmetry is not acknowledged.

The submission contains only two ablations (ScaleHead and DepthHead), both on HOT3D alone. Nothing tests L_coupled — the coupled supervision that is the paper's stated mechanism — and nothing tests the value of EgoX-Data-6M itself (e.g. training the same model without it).

Questions for authors

(a) What is the measured accuracy of EgoX-Engine annotations? Please report a validation against an independent ground-truth source (marker-based mocap, manual verification, or a public benchmark with trusted labels) for hand joints, object pose and metric scale, with error distributions rather than exemplars.

(b) Ablate L_coupled. What happens to each of the four tasks when the coupled term is removed? And what happens when EgoX-Data-6M is removed from the mixture?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.