- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper presents Exo2EgoHOI, a framework for translating exocentric manipulation videos into egocentric videos while preserving the observed hand–object interactions and object consistency. Built on a pretrained video diffusion model, the framework incorporates two key components. First, it uses depth estimation, hand reconstruction, and object segmentation to construct a unified 4D HOI prior that combines scene geometry, articulated hand poses, and hand–object relations. A dual-branch residual adapter injects hand structure and interaction cues into the generative backbone. Second, it introduces Decomposed Gated Cross-Attention, which combines global semantic and local appearance features through learned block-specific gates to maintain object identity and appearance throughout the generated videos.
Experiments show that the method outperforms the evaluated baselines on most metrics of visual fidelity, object consistency, and hand-pose consistency, while ablation studies support the effectiveness of the unified 4D HOI prior and Decomposed Gated Cross-Attention.
Critical strengths
-
The method is well motivated and addresses the task’s key challenges. The explicit modeling of hand–object relations in the proposed 4D HOI prior provides useful constraints for preserving fine-grained interactions. DGCA further strengthens object appearance conditioning by incorporating CLIP features from object regions. Both designs are reasonable and closely aligned with the goal of preserving interactions and object consistency across viewpoints.
-
The experiments demonstrate clear performance improvements. On ARCTIC-HOI, object mIoU increases by 32.3% and PA-MPJPE decreases by 50.0% relative to their respective strongest evaluated baselines, while the method also improves most evaluation metrics on Ego-Exo4D. Ablation studies are comprehensive and provide convincing evidence of the effectiveness of the individual components.
Critical weaknesses
- Generalization is insufficiently demonstrated. The experiments appear to rely on separate fine-tuning and evaluation for each dataset, providing limited evidence of transfer without target-domain adaptation. Although ARCTIC-HOI includes 60 in-the-wild test clips, their results are not reported separately, and the supplementary website presents only two examples from similar tabletop settings. It therefore remains unclear whether the reported gains persist across diverse unseen environments, objects, and interactions, and whether a single model can retain these benefits when jointly trained on heterogeneous data sources.
- The evaluation does not directly establish interaction preservation. The reported metrics primarily assess hand pose and object appearance/localization separately, providing only indirect evidence that their spatial and contact relationships are preserved. In particular, per-frame, per-hand Procrustes alignment removes transformations that can be essential to hand-object contact. A targeted evaluation of contact preservation, relative hand-object configuration, or interaction correctness would better substantiate the paper’s central claim.
Questions for authors
- Could you provide additional cross-domain and generalization evaluations, including results on the in-the-wild subset?
- Could you directly evaluate HOI preservation?
4 comments
Dear Reviewer b4P7,
Thank you again for your time and feedback. As the rebuttal period draws to a close, please let us know if you have any remaining questions or concerns about our paper or responses. We would be happy to provide further clarification and continue the discussion.
Best,
I appreciate the authors' thoughtful and comprehensive rebuttal and am satisfied with their responses. I maintain my acceptance recommendation.
Thank you very much for acknowledging our effort. We are glad that we were able to address your concerns.
Thank you once again for the positive feedback!
Official Comment by Authors
Thank you for the constructive review. Below is the detailed response for cross-domain generalization, direct HOI evaluation, and the interpretation of PA alignment.
W1 / Q1: Cross-domain generalization on the in-the-wild subset.
We provide the evaluation results of original ARCTIC-trained model on the 60-clip in-the-wild subset without additional fine-tuning. This evaluates transfer from the controlled ARCTIC setting to the collected real-world manipulation videos. The complete comparison is provided below.
Our method achieves the best results on nine of ten metrics, including all six object- and hand-consistency metrics; compared with the strongest baseline for each metric, mIoU increases from 0.4087 to 0.4598 and PA-MPJPE decreases from 15.76 to 6.79 mm. These fixed-checkpoint results support transfer to the evaluated in-the-wild subset without fine-tuning.
Corresponding revision: The complete in-the-wild comparison and its discussion have been added to Table 6 and Appendix A.5.
W2 / Q2: Direct evaluation of hand–object interaction preservation.
According to the ARCTIC Dataset, we evaluate reconstructed hand–object geometry and hand-object-interaction against ARCTIC’s official 3D annotations. Generated frames are reconstructed using frozen ArcticNet-SF with calibrated intrinsics and known object templates. Additional evaluation metrics include contact and interaction consistency through CDev, Contact F1, and FieldMAE, together with video motion fidelity through object trajectories (TrajErr), coordinated contact motion (MDev), and object articulation (AAE). CDev evaluates the separation of predicted hand–object vertex pairs corresponding to annotated contacts; Contact F1 measures vertex-level contact agreement at a 3 mm threshold; FieldMAE evaluates the hand-to-object distance field. TrajErr, MDev, and AAE assess object trajectories, coordinated hand–object movement during contact, and object articulation, respectively. Detailed definitions of the metrics are provided in the supplementary materials (Appendix. A3). The complete comparison is provided below.
The evaluation directly measures contact and interaction consistency, together with video motion fidelity. The complete results for these metrics are:
Our method achieves the best results on all six metrics. Compared with the strongest baseline for each metric, CDev decreases from 119.75 to 103.58 mm, Contact F1 increases from 4.74% to 5.61%, and FieldMAE decreases from 28.54 to 25.04 mm, supporting improved contact and relative spatial consistency. TrajErr, MDev, and AAE also decrease from 180.54 to 121.49 mm, 24.05 to 21.21 mm/frame, and 25.17° to 21.81°, respectively, supporting better preservation of demonstrated motion and interaction progression.
Corresponding revision: These results have been added to Table 2 and Sec. 4.2, with metric definitions in Appendix A.3.4–A.3.5.