- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes EgoXperience, a full-stack paradigm for jointly reconstructing camera pose, scene depth, two-hand MANO meshes, and rigid-object 6-DoF poses from monocular egocentric video in a shared metric world frame. It has three parts: (1) EgoX-Engine, an automatic annotation pipeline that combines stereo depth and SLAM poses from a head-mounted rig with FoundationPose tracking of pre-scanned object meshes, verified against depth and silhouettes; (2) EgoX-Data-6M, a large dataset with many objects, tasks, and second-level captions, plus a held-out EgoX-Bench; and (3) EgoX, a DINOv3-based transformer that predicts all quantities in a single forward pass and adds a geometrically coupled loss that composes hand and object predictions with the predicted camera in world space.
On HOT3D, EgoX-Bench, HOI4D, and H2O, EgoX ranks first on hand, object, and depth metrics, and the authors report large reductions in camera, hand, and depth errors and a clear gain in object pose scores over the strongest baselines. The authors acknowledge that VGGT-Ω and DA3 Nested are better on camera pose in several benchmarks.
Critical strengths
- Unique dataset coverage. Table 1 shows no existing dataset provides 3D hands, 6-DoF objects, meshes, depth, masks, and captions across single-area, cross-area, and outdoor settings; EgoX-Data-6M does.
- Scalable annotation without mocap. The stereo + SLAM + FoundationPose verification loop yields mutually aligned annotations without motion-capture infrastructure.
- Efficiency and unified output. 14.60 s for 150 frames vs. 134–167 s for multi-stage pipelines; EgoX is the only compared method producing all outputs end-to-end.
Critical weaknesses
- Train/eval overlap and asymmetric comparison. Appendix G lists HOT3D, HOI4D, H2O, HO-Cap, H2O-3D, and DexYCB in the training mix, and three of the four main benchmarks are among them. Baselines appear to be evaluated with off-the-shelf checkpoints, with no indication of equivalent in-domain fine-tuning. This undermines the "state-of-the-art" claim.
- Circular ground truth on EgoX-Bench. Labels come from the FoundationPose-based engine, yet FoundationPose is a baseline in Table 3. The model may be mimicking the labeler's biases, and the labels are never validated against independent mocap or manual annotation.
- No ablation of the core contribution. The coupled supervision loss (Eq. 2) is never ablated; only ScaleHead (Table 5) and DepthHead (Table 6) are. Architectural choices (block counts, token counts) are also unablated.
- Reproducibility deferred. Appendix G states loss weights, sampling probabilities, optimizer schedule, seeds, hardware, and training time will be reported only in the final release.
- Proprietary hardware dependence. The pipeline relies on a specific commercial rig and its processing stack, limiting the paradigm claim.
- Rigid, pre-scanned objects only. All 101 objects are pre-scanned in a dome; generalization to unseen instances/categories and to articulated or deformable objects is untested, while several baselines target novel objects.
Questions for authors
- Were the baselines (VGGT-Ω, HaWoR, MINT, ReViV, HandFlow, etc.) fine-tuned on the same HOT3D/HOI4D/H2O train splits? Can the SOTA claim be reproduced with a model trained on EgoX-Data-6M only, excluding these splits?
- Has the accuracy of EgoX-Bench pseudo-labels been validated against independent mocap or a manually verified subset? How do you rule out shared inductive bias with FoundationPose?
- What is the effect of removing the local, dense, coupled, and scale losses separately?
- What fraction of evaluation objects is unseen during training, and how does performance on unseen categories compare?
- The paper uses the term entity. What exactly does it refer to in the proposed method?
- Depth, scale, camera, hand, and object predictions appear to be produced separately. How consistent are these outputs when evaluated in a common coordinate system? For example, does the hand region in the depth map align with the predicted hand pose? It would be helpful to visualize and evaluate consistency across other combinations of outputs as well.
- More details on the training and evaluation protocols are needed. For training, please provide a table summarizing the size of each dataset, the annotations used, and how missing annotations are handled (e.g., H2O lacks object masks). For evaluation, please similarly detail the size of each dataset and the scenarios evaluated.
Thank you to the authors for the comprehensive rebuttal and the substantial additional experiments. I appreciate the effort taken to address my concerns.
Some minor limitations remain, particularly regarding the detailed reproducibility of the training configuration and the scope of the generalization claims. Nevertheless, I believe the authors have substantially addressed the main concerns raised in my original review.
Overall, the additional experiments significantly strengthen the paper. I am therefore inclined to raise my assessment, as the proposed dataset, annotation pipeline, and unified reconstruction framework are supported by considerably stronger experimental evidence after the rebuttal.