acceptodds
Under review as a conference paper at ICLR 2027

Exo2EgoHOI: Hand-Object-Interaction Aware Exocentric-to-Egocentric Video Generation

Abstract

Egocentric videos of human manipulation provide valuable visual experience for embodied intelligence, yet collecting such data at scale is costly. Exocentric-to-egocentric video generation offers a scalable alternative by transforming abundant third-person manipulation videos into first-person observations. However, existing methods often struggle to faithfully preserve demonstrated hand-object interactions (HOI) across large viewpoint changes due to insufficient fine-grained interaction guidance and weak object-centric anchoring. We present Exo2EgoHOI, an HOI-aware video generative framework for interaction-preserving exocentric-to-egocentric translation. To preserve fine-grained HOI, we introduce a unified 4D HOI prior that combines scene geometry, articulated hand renderings, and dense hand-object relation fields, together with a dual-branch residual adapter for injecting structural and relational cues into the video generation backbone. To preserve object consistency, we introduce Decomposed Gated Cross-Attention, which separately encodes object and background references and adaptively integrates global semantic and local appearance features as object-centric anchors. Experiments on ARCTIC-HOI and Ego-Exo4D demonstrate consistent improvements in visual fidelity, object consistency, and HOI preservation. In particular, on ARCTIC-HOI, Exo2EgoHOI improves object mIoU by 32.3% and reduces MPJPE and PA-MPJPE by 34.7% and 50.0%, respectively, over the strongest baseline for each metric. The code will be released to benefit the research community.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Exocentric-to-egocentric video generation, hand–object interaction

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_b4P7OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper presents Exo2EgoHOI, a framework for translating exocentric manipulation videos into egocentric videos while preserving the observed hand–object interactions and object consistency. Built on a pretrained video diffusion model, the framework incorporates two key components. First, it uses depth estimation, hand reconstruction, and object segmentation to construct a unified 4D HOI prior that combines scene geometry, articulated hand poses, and hand–object relations. A dual-branch residual adapter injects hand structure and interaction cues into the generative backbone. Second, it introduces Decomposed Gated Cross-Attention, which combines global semantic and local appearance features through learned block-specific gates to maintain object identity and appearance throughout the generated videos.

Experiments show that the method outperforms the evaluated baselines on most metrics of visual fidelity, object consistency, and hand-pose consistency, while ablation studies support the effectiveness of the unified 4D HOI prior and Decomposed Gated Cross-Attention.

Critical strengths
  • The method is well motivated and addresses the task’s key challenges. The explicit modeling of hand–object relations in the proposed 4D HOI prior provides useful constraints for preserving fine-grained interactions. DGCA further strengthens object appearance conditioning by incorporating CLIP features from object regions. Both designs are reasonable and closely aligned with the goal of preserving interactions and object consistency across viewpoints.

  • The experiments demonstrate clear performance improvements. On ARCTIC-HOI, object mIoU increases by 32.3% and PA-MPJPE decreases by 50.0% relative to their respective strongest evaluated baselines, while the method also improves most evaluation metrics on Ego-Exo4D. Ablation studies are comprehensive and provide convincing evidence of the effectiveness of the individual components.

Critical weaknesses
  • Generalization is insufficiently demonstrated. The experiments appear to rely on separate fine-tuning and evaluation for each dataset, providing limited evidence of transfer without target-domain adaptation. Although ARCTIC-HOI includes 60 in-the-wild test clips, their results are not reported separately, and the supplementary website presents only two examples from similar tabletop settings. It therefore remains unclear whether the reported gains persist across diverse unseen environments, objects, and interactions, and whether a single model can retain these benefits when jointly trained on heterogeneous data sources.
  • The evaluation does not directly establish interaction preservation. The reported metrics primarily assess hand pose and object appearance/localization separately, providing only indirect evidence that their spatial and contact relationships are preserved. In particular, per-frame, per-hand Procrustes alignment removes transformations that can be essential to hand-object contact. A targeted evaluation of contact preservation, relative hand-object configuration, or interaction correctness would better substantiate the paper’s central claim.
Questions for authors
  • Could you provide additional cross-domain and generalization evaluations, including results on the in-the-wild subset?
  • Could you directly evaluate HOI preservation?
4 comments
AuthorsOpenReview ↗

Official Comment by Authors

Thank you for the constructive review. Below is the detailed response for cross-domain generalization, direct HOI evaluation, and the interpretation of PA alignment.

W1 / Q1: Cross-domain generalization on the in-the-wild subset.

We provide the evaluation results of original ARCTIC-trained model on the 60-clip in-the-wild subset without additional fine-tuning. This evaluates transfer from the controlled ARCTIC setting to the collected real-world manipulation videos. The complete comparison is provided below.

MethodPSNR ↑ (dB)SSIM ↑LPIPS ↓CLIP-I ↑mIoU ↑Center-Err. ↓CLIP-O ↑MPJPE ↓ (mm)WA-MPJPE ↓ (mm)PA-MPJPE ↓ (mm)
WAN VACE18.290.55840.54070.83530.16690.12150.81362311.39186.6818.83
EgoWorld18.600.54490.32610.84810.29570.09120.80481413.57198.1415.76
Vista4D17.130.59350.41160.89770.40870.06040.83632180.77205.3815.81
EgoX19.440.59010.41600.90670.39750.06240.83922123.13175.8316.60
Ours19.470.65760.34490.91160.45980.03810.87541368.65164.696.79

Our method achieves the best results on nine of ten metrics, including all six object- and hand-consistency metrics; compared with the strongest baseline for each metric, mIoU increases from 0.4087 to 0.4598 and PA-MPJPE decreases from 15.76 to 6.79 mm. These fixed-checkpoint results support transfer to the evaluated in-the-wild subset without fine-tuning.

Corresponding revision: The complete in-the-wild comparison and its discussion have been added to Table 6 and Appendix A.5.

W2 / Q2: Direct evaluation of hand–object interaction preservation.

According to the ARCTIC Dataset, we evaluate reconstructed hand–object geometry and hand-object-interaction against ARCTIC’s official 3D annotations. Generated frames are reconstructed using frozen ArcticNet-SF with calibrated intrinsics and known object templates. Additional evaluation metrics include contact and interaction consistency through CDev, Contact F1, and FieldMAE, together with video motion fidelity through object trajectories (TrajErr), coordinated contact motion (MDev), and object articulation (AAE). CDev evaluates the separation of predicted hand–object vertex pairs corresponding to annotated contacts; Contact F1 measures vertex-level contact agreement at a 3 mm threshold; FieldMAE evaluates the hand-to-object distance field. TrajErr, MDev, and AAE assess object trajectories, coordinated hand–object movement during contact, and object articulation, respectively. Detailed definitions of the metrics are provided in the supplementary materials (Appendix. A3). The complete comparison is provided below.

The evaluation directly measures contact and interaction consistency, together with video motion fidelity. The complete results for these metrics are:

MethodCDev ↓ (mm)Contact F1 ↑ (%)FieldMAE ↓ (mm)TrajErr ↓ (mm)MDev ↓ (mm/frame)AAE ↓ (degrees)
WAN VACE168.682.6737.71180.5430.7042.08
EgoWorld157.852.4635.01355.88124.2358.33
Vista4D169.152.8435.36245.7235.4737.04
EgoX119.754.7428.54199.6124.0525.17
Ours103.585.6125.04121.4921.2121.81

Our method achieves the best results on all six metrics. Compared with the strongest baseline for each metric, CDev decreases from 119.75 to 103.58 mm, Contact F1 increases from 4.74% to 5.61%, and FieldMAE decreases from 28.54 to 25.04 mm, supporting improved contact and relative spatial consistency. TrajErr, MDev, and AAE also decrease from 180.54 to 121.49 mm, 24.05 to 21.21 mm/frame, and 25.17° to 21.81°, respectively, supporting better preservation of demonstrated motion and interaction progression.

Corresponding revision: These results have been added to Table 2 and Sec. 4.2, with metric definitions in Appendix A.3.4–A.3.5.

Dear Reviewer b4P7,

Thank you again for your time and feedback. As the rebuttal period draws to a close, please let us know if you have any remaining questions or concerns about our paper or responses. We would be happy to provide further clarification and continue the discussion.

Best,

I appreciate the authors' thoughtful and comprehensive rebuttal and am satisfied with their responses. I maintain my acceptance recommendation.

Thank you very much for acknowledging our effort. We are glad that we were able to address your concerns.

Thank you once again for the positive feedback!

Reviewer_HRUEOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Exo2EgoHOI, a framework for generating egocentric videos from exocentric human manipulation demonstrations while preserving hand-object interactions. The method introduces a unified 4D HOI prior that combines scene geometry, hand articulation, and hand-object relations, together with a decomposed gated cross-attention mechanism for maintaining object consistency. Experiments on ARCTIC-HOI and Ego-Exo4D demonstrate improvements in visual quality and several interaction-related metrics.

Critical strengths

S1.The unified 4D HOI prior explicitly incorporates hand articulation and hand-object spatial relationships beyond scene-level geometry. This provides a reasonable approach to preserving fine-grained interaction information across large viewpoint changes.

S2.The method achieves substantial improvements in object consistency and hand-pose accuracy on ARCTIC-HOI. The ablation studies further demonstrate the complementary contributions of scene, hand, and interaction priors, as well as the proposed reference-conditioning mechanism.

Critical weaknesses

W1.The central contribution focuses on preserving hand-object interactions, but the evaluation primarily relies on visual similarity, object consistency, and hand-pose estimation metrics. These metrics do not directly establish whether the generated videos preserve contact dynamics and interaction consistency.

Questions for authors

Q1.Can the authors provide more direct evidence that the generated videos preserve hand-object interactions, rather than only improving hand-pose and object appearance metrics?

Q2.Why does the method show a considerable degradation in PA-MPJPE on Ego-Exo4D, despite achieving strong improvements on ARCTIC-HOI?

2 comments
AuthorsOpenReview ↗

Official Comment by Authors

Thank you for the constructive comments. Below is the detailed response for direct evaluation of interaction preservation and the PA-MPJPE degradation on Ego-Exo4D.

W1 / Q1: Direct evaluation of contact preservation and video-level interaction consistency.

According to the ARCTIC Dataset, we evaluate reconstructed hand–object geometry and hand-object-interaction against ARCTIC’s official 3D annotations. Generated frames are reconstructed using frozen ArcticNet-SF with calibrated intrinsics and known object templates.

Additional evaluation metrics include contact and interaction consistency through CDev, Contact F1, and FieldMAE, together with video motion fidelity through object trajectories (TrajErr), coordinated contact motion (MDev), and object articulation (AAE). CDev evaluates the separation of predicted hand–object vertex pairs corresponding to annotated contacts; Contact F1 measures vertex-level contact agreement at a 3 mm threshold; FieldMAE evaluates the hand-to-object distance field. TrajErr, MDev, and AAE assess object trajectories, coordinated hand–object movement during contact, and object articulation, respectively. Detailed definitions of the metrics are provided in the supplementary materials (Appendix. A3).

The complete comparison is provided below.

MethodMPJPE ↓ (mm)WA-MPJPE ↓ (mm)PA-MPJPE ↓ (mm)CDev ↓ (mm)Contact F1 ↑ (%)FieldMAE ↓ (mm)TrajErr ↓ (mm)MDev ↓ (mm/frame)AAE ↓ (degrees)
WAN VACE1443.3673.5522.45168.682.6737.71180.5430.7042.08
EgoWorld2820.98172.9874.79157.852.4635.01355.88124.2358.33
Vista4D1549.23114.9541.04169.152.8435.36245.7235.4737.04
EgoX1242.1983.7925.65119.754.7428.54199.6124.0525.17
Ours1187.3262.7513.79103.585.6125.04121.4921.2121.81

Our method achieves the best results on all nine metrics. Compared with the strongest baseline for each metric, CDev decreases from 119.75 to 103.58 mm, Contact F1 increases from 4.74% to 5.61%, and FieldMAE decreases from 28.54 to 25.04 mm. These interaction metrics are computed without fitted spatial alignment, providing evidence of improved contact and relative hand–object geometry.

For video motion fidelity, TrajErr decreases from 180.54 to 121.49 mm, MDev from 24.05 to 21.21 mm/frame, and AAE from 25.17° to 21.81°. Together, these improvements support better preservation of demonstrated object motion, coordinated hand–object movement during contact, and object articulation.

Corresponding revisions: These results have been added to Table 2 and Sec. 4.2, with metric definitions in Appendix A.3.4–A.3.5.

Q2: Why is PA-MPJPE worse on Ego-Exo4D?

Our method achieves the best reported MPJPE (5208.17 mm) and WA-MPJPE (1749.36 mm) on Ego-Exo4D, although its PA-MPJPE is higher than those of WAN VACE and EgoX. Ego-Exo4D covers diverse activities beyond close-up hand–object manipulation, and hands in the selected clips can occupy small image regions or be blurred and heavily occluded. These conditions can reduce the reliability of WiLoR estimates and introduce finger-articulation errors into the hand and interaction priors. Since our model explicitly conditions on these priors, such errors may propagate into the generated hand configurations, affecting fine-grained finger details. PA-MPJPE removes overall translation, rotation, and scale differences, so these residual configuration errors remain visible after alignment, even when the unaligned and sequence-aligned errors are lower.

Dear Reviewer HRUE,

Thank you again for your time and feedback. As the rebuttal period draws to a close, please let us know if you have any remaining questions or concerns about our paper or responses. We would be happy to provide further clarification and continue the discussion.

Best,

Reviewer_TSAJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents Exo2EgoHOI for exocentric-to-egocentric video generation with an emphasis on preserving hand-object interactions. Its main contibutions are a unified 4D HOI prior combining scene geometry, hand renderings, hand-object relation fields, and Decomposed Gated Cross-Attention, which seperated object/background references and combines global and local appearance features. The approach adapts a pretrained video diffusion model (WAN) and introduces an interaction-oriented benchmark derived from ARCTIC, supplemented with in-the-wild clips. Experiments on ARCTIC-HOI and Ego-Exo4D show improvements in most reported metrics, with substantially larger gains on ARCTIC-HOI. Component-wise ablations examine the geometric priors, reference conditioning, and training-view diversity.

Critical strengths
  1. A coherent framework for preserving the functional content of manipulation across viewpoints. The paper presents a coherent framework for preserving the functional content of manipulation across viewpoints. It identifies the complementary challenges of maintaining fine-grained hand object geometry and preserving the manipulated object's identity, appearance, and state. The proposed scene-hand-interaction prior and object-focused reference conditioning directly address these challenges. Explicitly representing hand-object proximity and direction provides relational information beyond the pose, while DGCA supplies appearance anchors where geometric reconstruction offers incomplete evidence. This combination is meaningful because faithful manipulation vieo translation requires spatial interaction structure and object consistency to be maintained together.

  2. A task-oriented evaluation framework with substantial component-wise analysis The paper develops an benchmark derived from ARCTIC supplemented with in-the-wild clips and evaluates visual fidelity, object consistency, and hand pose consistency across two datasets. This evaluation captures several important dimensions of manipulation video generation. The experiments examine the geometric priors, global and local reference features, learned fusion, and training-view diversity. Together with qualitative comparisons and failure cases, these studies provide useful evidence about which design choices contribute to preserving manipulation content and where the approach remains limited. The broad improvements on ARCTIC-HOI provide a substantive empirical basis for the proposed framework.

Critical weaknesses
  1. Reference hand poses depend on an estimator Hand metrics compare WiLoR predictions on reference and generated images, making it difficult to distinguish generation errors from hand pose estimation errors on both images. Without evaluation against ARCTIC's annotations, it remains unclear whether the reported improvements also hold for actual hand pose accuracy.

  2. Hand pose improvements are not consistent across environments On Ego-Exo4D, PA-MPJPE is worse than both WAN VACE and EgoX at 93.81 versus 74.00 and 75.55, as acknowledged in Section 4.2. This limits the evidence that the improvements in fine-grained hand articulation observed on ARCTIC-HOI generalize to more diverse environments.

  3. Temporal video quality is insufficiently evaluated Most reported metrics assess frame-level appearance, object placement, or hand pose agreement. Although WA-MPJPE uses sequence-level alignment, the paper does not provide a dedicated evaluation of whether the generated videos faithfully preserve the demonstrated object motion and action progression over time.

  4. The scope of the robot demonstration is unclear Section 5 identifies downstream embodied learning and robot manipulation as future work. However, approximately one-third of Figure 1 is devoted to "Real Robot Execution" suggesting that some downstream some downstream capability has already been demonstrated. The paper does not explain whether this is a preliminary proof of concept using the generated videos or an illustration of a future application. No action-extraction procedure, retargeting method, robot setup, or manual intervention is documented. The reported hand pose metrics alone also do not establish suitability for successful retargeting, leaving the scope and supporting evidence for the prominently featured demonstration unclear.

Questions for authors
  1. Reference hand poses Why is WiLoR applied to reference images instead of using ARCTIC's annotated 3D geometry? Given that WiLoR also constructs the input hand priors, can the authors verify whether the improvements persist under annotation-based evaluation?

  2. Viewpoint selection Appendix A.2 states that cameras 2 and 7 are excluded because their viewpoints differ substantially from the target egocentric view. Could the authors clarify how this difference was assessed and quantify the viewpoint range covered by the benchmark? Results on the excluded views would help establish the scope of the claimed robustness to large viewpoint changes.

  3. Robot execution pipeline Was the robot execution in Figure 1 driven by actions extracted from the generated videos? Please describe the robot setup, pipelines, and any experimental setups.

  4. Feasibility of contact evaluation If feasible, could the authors provide a simple quantitative or qualitative analysis of whether hand object contact relationships are preserved?

  5. Video-level motion fidelity Could the authors provide a video-level evaluation of whether the generated videos preserve the demonstrated object motion and action progression? While WA-MPJPE uses sequence-level alignment, an additional assessment of motion fidelity would help establish whether the reported frame-level improvements translate into temporally coherent manipulation videos.

7 comments
AuthorsOpenReview ↗

Official Comment by Authors (Part 1)

Thank you for the detailed and constructive review. Below is the detailed responses.

W1 / Q1: Hand evaluation against ARCTIC’s 3D annotations.

The original evaluation uses WiLoR on both reference and generated frames to provide a common protocol across the evaluation sources. We agree that this introduces reference-estimation uncertainty. Following the setting in ARCTIC, we therefore additionally evaluate generated hands against ARCTIC’s official 3D hand annotations:

MethodMPJPE ↓ (mm)WA-MPJPE ↓ (mm)PA-MPJPE ↓ (mm)
WAN VACE1443.3673.5522.45
EgoWorld2820.98172.9874.79
Vista4D1549.23114.9541.04
EgoX1242.1983.7925.65
Ours1187.3262.7513.79

Our method achieves the best results on all three hand metrics. Compared with the strongest baseline for each metric, MPJPE decreases from 1242.19 to 1187.32 mm, WA-MPJPE from 73.55 to 62.75 mm, and PA-MPJPE from 22.45 to 13.79 mm. These results confirm that the hand-pose improvements persist when the reference is annotated geometry rather than estimated reference poses.

Corresponding revision: The annotation-based hand results have been added to Table 2 and Sec. 4.2.

W2: Why is PA-MPJPE worse on Ego-Exo4D?

Our method achieves the best reported MPJPE (5208.17 mm) and WA-MPJPE (1749.36 mm) on Ego-Exo4D, although its PA-MPJPE is higher than those of WAN VACE and EgoX. Ego-Exo4D covers diverse activities beyond close-up hand–object manipulation, and hands in the selected clips can occupy small image regions or be blurred and heavily occluded. These conditions can reduce the reliability of WiLoR estimates and introduce finger-articulation errors into the hand and interaction priors. Since our model explicitly conditions on these priors, such errors may propagate into the generated hand configurations, affecting fine-grained finger details. PA-MPJPE removes overall translation, rotation, and scale differences, so these residual configuration errors remain visible after alignment, even when the unaligned and sequence-aligned errors are lower.

Q2: Source-camera selection and robustness to cameras 2 and 7.

Cameras 2 and 7 observe the interaction from low, upward-looking viewpoints, whereas the egocentric camera primarily looks downward. Consequently, they share limited visible interaction content with the egocentric view: the manipulated object and contact region can be heavily occluded. This reduces the source-view information available for reconstructing the HOI prior. So, we didn't include camera 2 and camera 7. We visualize some samples (Figure. 7 in Appendix) from camera 2 and camera 7 to illustrate these visibility limitations.

We also construct 200 additional test clips in total using cameras 2 and 7 as source views. We compare all methods on this additional set, directly evaluating our original trained model without fine-tuning. The evaluation uses ARCTIC’s official 3D annotations as references.

MethodMPJPE ↓ (mm)WA-MPJPE ↓ (mm)PA-MPJPE ↓ (mm)CDev ↓ (mm)Contact F1 ↑ (%)FieldMAE ↓ (mm)TrajErr ↓ (mm)MDev ↓ (mm/frame)AAE ↓ (degrees)
WAN VACE1680.40473.4420.18161.173.9836.31159.2411.2343.03
EgoWorld4351.34613.3487.70166.772.3838.15323.39104.3858.56
Vista4D1678.19499.6634.28156.283.9232.95259.8623.9946.00
EgoX1870.64465.1750.17156.634.0232.63217.0015.4348.88
Ours1533.91403.9526.40137.435.5131.78146.129.0929.97

Our method achieves the best results on eight of nine metrics, including all contact/spatial-consistency and video-motion metrics, supporting its robustness to these source cameras excluded from training.

Corresponding revisions: The visibility examples and additional-view results have been added to Fig. 7, Table 5, and Appendix A.2; Fig. 6 shows the camera configuration.

AuthorsOpenReview ↗

Official Comment by Authors (Part 2)

W4 / Q3: Real-robot setup and execution pipeline.

The demonstration is a real-robot proof of concept driven by actions reconstructed from our generated egocentric videos.

We use a Unitree G1 humanoid mounted on a fixed support stand and equipped with an Inspire RH56E2 dexterous hand with six actuated degrees of freedom. The arm and waist are joint-position controlled at 500 Hz, and the hand at 50 Hz. The pipeline is:

  1. Video generation and HOI reconstruction. The robot’s onboard camera captures an exocentric human demonstration, which our method translates into an egocentric video. Following the reconstruction stage of Do-As-I-Do [1], we recover per-frame 3D hand geometry, an object mesh, and 6-DoF object poses from the generated video.

  2. Workspace registration and retargeting. A single rigid transformation registers the reconstructed interaction to the robot tabletop workspace while preserving the relative hand–object configuration. We adapt kinematic retargeting to the Inspire hand using human fingertip targets expressed in the palm frame. Collision-aware arm inverse kinematics tracks the palm pose relative to the object under joint limits, table/self-collision constraints, and temporal smoothness.

  3. Simulation and transfer to hardware. We simulate the retargeted robot motion in MuJoCo before transferring the resulting joint trajectory to the physical robot.

  4. Real-world execution. Physical objects are placed at their registered initial locations. From its measured state, the robot plans a collision-checked approach to the first retargeted configuration and then replays the trajectory under joint-position control without online visual or force feedback.

Corresponding revisions: The robot setup and pipeline are documented in Sec. 4.2 and Appendix A.4, with the generation, simulation, and real-world execution stages visualized in Fig. 8.

[1] Do as I Do: Dexterous Manipulation Data from Everyday Human Videos, https://arxiv.org/abs/2606.19333.

Q4 and W3 / Q5: Direct contact evaluation and video-level motion fidelity.

Using the annotation-based geometry evaluation described above, we additionally assess contact and relative spatial consistency, together with video motion fidelity. Generated hand–object geometry is compared against ARCTIC’s official 3D annotations using the following six metrics (metric definitions are provided in Appendix A.3.4–A.3.5.):

MethodCDev ↓ (mm)Contact F1 ↑ (%)FieldMAE ↓ (mm)TrajErr ↓ (mm)MDev ↓ (mm/frame)AAE ↓ (degrees)
WAN VACE168.682.6737.71180.5430.7042.08
EgoWorld157.852.4635.01355.88124.2358.33
Vista4D169.152.8435.36245.7235.4737.04
EgoX119.754.7428.54199.6124.0525.17
Ours103.585.6125.04121.4921.2121.81

CDev measures the separation of predicted vertex pairs corresponding to annotated contacts; Contact F1 evaluates vertex-level contact at a 3 mm threshold; FieldMAE measures hand-to-object distance-field error.

For temporal evaluation, TrajErr compares time-corresponding object positions in world coordinates, MDev measures coordinated hand–object movement within annotated stable-contact windows, and AAE compares object-articulation angles. Together, these metrics assess demonstrated trajectories and interaction progression beyond frame-level appearance.

Our method achieves the best results on all six metrics. Compared with the strongest baseline for each metric, CDev decreases from 119.75 to 103.58 mm, Contact F1 increases from 4.74% to 5.61%, and FieldMAE decreases from 28.54 to 25.04 mm. TrajErr, MDev, and AAE also decrease from 180.54 to 121.49 mm, 24.05 to 21.21 mm/frame, and 25.17° to 21.81°, respectively. These results support improved contact/spatial consistency and motion preservation.

Corresponding revisions: The contact and motion results have been added to Table 2 and Sec. 4.2, with metric definitions in Appendix A.3.4–A.3.5.

Thank you for the detailed response and additional experiments. The annotation-based hand evaluation, camera 2/7 results, and contact/motion metrics address several of my concerns. Two main concerns remain:

1. Fine-grained hand articulation. PA-MPJPE, the metric most directly tied to the paper's central claim, is consistently not the best under harder conditions (Ego-Exo4D: 93.81 vs. 74.00; cameras 2/7: 26.40 vs. 20.18). Your W2 response attributes this to WiLoR errors propagating through the priors, which indicates a direct dependence on the input hand estimator. Please state this as a limitation.

2. Robot demonstration. I appreciate the clarified pipeline, but some concerns remain. Since the method already reconstructs a 4D HOI from the exocentric input, the benefit of retargeting from the generated egocentric video, rather than from this reconstruction directly, remains unclear to me. In addition, given the contact accuracy (CDev ≈ 10 cm, Contact F1 ≈ 5.6%) and the absence of a success rate or policy-learning results, the evidence that the generated videos are useful for robot learning is still limited. I view the demonstration as a qualitative proof of concept rather than support for downstream utility.

Reviewer_kGYcOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Exo2EgoHOI translates exocentric videos into egocentric videos using scene geometry, hand meshes and hand-object relation fields. Built on Wan2.1 with LoRA, it combines a dual-branch HOI adapter with Decomposed Gated Cross-Attention over object/background references. Experiments on ARCTIC-HOI and Ego-Exo4D report improvements in object consistency and several hand-pose metrics, supported by component ablations.

Critical strengths
  1. The framework combines geometric, interaction and appearance cues to address a meaningful limitation of cross-view video generation.

  2. The reported ARCTIC-HOI results are strong: object mIoU improves from 0.2933 to 0.3881 and PA-MPJPE decreases from 20.36 to 10.18 mm relative to the best baseline for each metric. Component ablations support the proposed conditioning design.

Critical weaknesses
  1. The rationale and effects of DGCA could be explained more clearly. The segmentation-based separation of foreground and background features needs a clearer explanation of how it improves consistency, supported by experimental evidence. The effect of the fusion weights also warrants further investigation.
Questions for authors
  1. Could you explain the rationale and consistency benefits of segmentation-based foreground/background separation in DGCA, with supporting evidence and further analysis of the fusion weights?
2 comments
AuthorsOpenReview ↗

Official Comment by Authors

Thank you for the constructive comments. Below is the detailed response.

W1 / Q1: Rationale and consistency benefits of regional decomposition.

The manipulated object often occupies only a small region of the exocentric view. Encoding the entire frame at a fixed resolution can therefore underrepresent object-specific details. A dedicated object crop gives the object a larger effective spatial representation, while the object-masked background retains complementary scene context. The shared CLIP encoder extracts a [CLS] token and patch tokens from each reference. The object and background [CLS] tokens form the global context, while their patch tokens form the local context (Eq. 11). Thus, both contexts contain object and background information. Together, these decomposed references help maintain both object and background consistency throughout video generation.

The ablation table provides supporting evidence for the complete reference-conditioning design:

Reference conditioningLPIPS ↓CLIP-I ↑Object mIoU ↑CLIP-O ↑MPJPE ↓ (mm)WA-MPJPE ↓ (mm)PA-MPJPE ↓ (mm)
Full-frame Vanilla CA (#8)0.41670.91080.36180.84881598.81320.6911.39
Decomposed anchors with fixed fusion (#7)0.40050.91560.33540.84791638.83322.4413.61
Full DGCA with learned fusion (#9)0.38600.91750.38810.85571318.74313.7510.18

The comparison with #8 supports the effectiveness of the complete DGCA conditioning design, while the comparison between #7 and #9 demonstrates the benefit of learning the global–local fusion weights. We have clarified the design rationale and the interpretation of these comparisons in Secs. 3.3 and 4.3.

Dear Reviewer kGYc,

Thank you again for your time and feedback. As the rebuttal period draws to a close, please let us know if you have any remaining questions or concerns about our paper or responses. We would be happy to provide further clarification and continue the discussion.

Best,

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions

Summary of Revisions

We thank the reviewers for their constructive feedback. The main revisions are summarized below:

  1. Clarified DGCA’s motivation and mechanism. We expanded the explanation of object/background reference decomposition and global–local fusion, and clarified the interpretation of the supporting ablations. Location: Secs. 3.3 and 4.3; Table 3.

  2. Added evaluation against ARCTIC’s 3D annotations. We added annotation-based hand evaluation, together with direct measurements of contact/spatial consistency and video motion fidelity. Location: Secs. 4.1–4.2; Table 2; Appendix A.3.4–A.3.5.

  3. Documented the real-robot execution pipeline. We added the hardware setup, HOI reconstruction from generated videos, retargeting, simulation, and real-world replay procedure, clarifying the demonstration’s preliminary scope. Location: Sec. 4.2; Appendix A.4; Fig. 8.

  4. Clarified camera selection and added viewpoint-robustness tests. We provided examples explaining the limited interaction visibility from cameras 2 and 7, and evaluated different methods on 200 additional clips from these cameras, using our original model without fine-tuning. Location: Appendix A.2; Fig. 7; Table 5.

  5. Added separate in-the-wild results. We reported quantitative comparisons on the 60-clip in-the-wild subset without additional fine-tuning to assess transfer beyond the controlled setting. Location: Appendix A.5; Table 6.

  6. Clarified the limitations of estimated HOI priors. We note that errors in upstream model predictions can compromise HOI prior accuracy and affect fine-grained hand articulation in challenging scenes. Location: Sec. 5, “Limitations and future work.”

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.