- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Any2Ego, a framework for egocentric viewpoint translation from third-person videos. The method integrates predicted hand–object interaction (HOI) masks with a learned first-person textual anchor to jointly capture spatial constraints and semantic information. Specifically, a vision-language model is trained through supervised caption distillation and generation-grounded reinforcement learning to produce structured egocentric descriptions. These descriptions are subsequently incorporated into a video diffusion model through HOI-conditioned spatial routing, which adaptively modulates the contribution of textual information at different spatial locations.
Critical strengths
- A meaningful integration of spatial and semantic conditioning. The paper addresses an important limitation of existing egocentric viewpoint translation methods by complementing HOI-based spatial constraints with learned first-person semantic descriptions. While HOI masks provide structural guidance for hand–object interactions, they cannot fully characterize viewpoint-dependent appearance, scene layout, and contextual details. The proposed textual anchor provides complementary semantic information, while HOI-conditioned spatial routing offers a reasonable mechanism for adaptively controlling its influence across different video regions. This integration is well motivated and practically relevant to egocentric video generation.
- Convincing empirical improvements supported by controlled ablation studies. Any2Ego consistently outperforms the reproduced EgoExo-Gen baseline across multiple evaluation metrics, demonstrating improvements in structural similarity, perceptual fidelity, and video distribution quality. In particular, the controlled comparison between global and spatial caption injection in Table 3 provides meaningful evidence for the effectiveness of HOI-conditioned routing. The additional analysis of residual motion fidelity further strengthens the experimental validation by demonstrating improvements in temporal consistency beyond frame-level appearance similarity. These results provide reasonable empirical support for the proposed framework.
Critical weaknesses
- Limited methodological novelty and insufficient mechanistic justification. Although the proposed framework demonstrates promising empirical improvements, its principal components are largely based on established techniques, including vision-language model distillation, group relative policy optimization (GRPO), cross-attention conditioning, and spatially adaptive gating. The main contribution therefore appears to lie in the integration and adaptation of existing mechanisms rather than the introduction of a fundamentally new modeling principle. While such integration can be valuable, the paper does not sufficiently articulate the distinctive methodological insight that makes the proposed approach more than a combination of complementary conditioning strategies. Furthermore, the theoretical analysis of spatial routing primarily characterizes the limitations of global response scaling under simplified assumptions. Although this analysis provides an intuitive motivation for spatial modulation, it does not directly explain how the learned HOI-conditioned gates achieve effective semantic localization or why they should outperform alternative spatial conditioning mechanisms.
- Limited evidence of generalization and substantial dependence on strong initial conditions. The experimental evaluation is restricted to short cooking videos from Ego-Exo4D, with each generated sequence containing only 13 frames. Although this setting provides a controlled benchmark for egocentric viewpoint translation, it does not adequately establish the applicability of the proposed framework to longer video sequences, more diverse activities, or substantially different interaction scenarios. More importantly, Any2Ego requires the ground-truth egocentric first frame and its corresponding HOI mask as input. These conditions already provide substantial information about the target viewpoint, scene appearance, and initial interaction geometry, potentially reducing the difficulty of the viewpoint translation task. While this first-frame-conditioned formulation is explicitly defined in the paper and follows the adopted evaluation protocol, it remains unclear how effectively the proposed framework would operate when such strong target-view supervision is unavailable or imperfect.
Questions for authors
- Could the authors further clarify the methodological distinction between the proposed HOI-conditioned spatial routing and existing spatially adaptive conditioning mechanisms? In particular, what evidence demonstrates that the learned routing gates provide meaningful semantic localization beyond conventional spatial feature modulation?
- How does Any2Ego generalize to longer video sequences or activities beyond cooking? Additionally, how sensitive is the framework to the availability and accuracy of the initial egocentric frame and HOI mask? An evaluation under imperfect or reduced initial-view conditioning would help clarify its practical applicability.
Response to Reviewer 5o5M: Spatial Routing, Generalization, and Input Conditions
Thank you for recognizing the motivation for combining spatial and semantic conditions and the value of our controlled ablations and residual-motion analysis. We clarify the contribution, generalization evidence, and practical input requirements below.
Q1: What distinguishes HOI-conditioned spatial routing, and what evidence supports semantic localization?
Our design connects two complementary conditions for first-frame-conditioned exo-to-ego generation: HOI prediction supplies interaction structure, while a learned first-person caption supplies appearance and scene information absent from structural labels. Spatial routing uses spatial-conditioning features to regulate the caption contribution at each video token.
Caption attention uses video queries and separate caption keys and values. A per-token sigmoid gate reads detached VACE conditioning features and, together with a learned per-block gain, scales the caption-attention response. This response is combined with native narration attention before the shared output projection. The trained HOI-conditioned base remains frozen, retaining the existing spatial and narration pathways (Eq. 6; Appendix I.5).
The matched Global-Spatial comparison provides direct evidence. Both use the same frozen base, captions, caption-projection architecture, and training schedule; Spatial adds one projection vector and one bias per main DiT block. Spatial improves FVD from 37.8 to 33.3, SSIM from 0.636 to 0.660, and LPIPS from 0.163 to 0.144 (Table 3). Residual correlation improves from 0.614 to 0.647 and residual PSNR from 20.47 to 21.17 dB (Table 7). These results support location-dependent caption modulation, which we interpret as interaction-conditioned spatial modulation rather than explicit word-to-region alignment.
Q2: What evidence supports generalization beyond cooking?
We added an evaluation on 80 fixed non-cooking ENIGMA-360 validation clips from 29 recordings, covering industrial activities. All methods use the same clips and metric implementation, without ENIGMA-360 fine-tuning (Appendix H, Table 8).
All configurations use exo video or exo-derived conditions; all except CogVideoX also receive the initial ego frame. Training histories differ, so this evaluates the specified configurations rather than architecture alone under matched training.
Any2Ego achieves the best mean on all four metrics, supporting transfer from cooking to industrial activities without target-dataset fine-tuning. FVD comparisons are within this common 80-clip evaluation; absolute values should not be directly compared with the 1,000-clip Ego-Exo4D results because sample count and video distribution differ.
Q3: Does dependence on the initial ego frame and HOI conditions limit practical applicability?
We agree that these conditions provide important information. However, the subsequent ego HOI sequence is predicted from the exocentric video and initial conditions, without collecting or annotating the complete target ego video. Exocentric video is already an input to the task: subsequent HOI conditions are generated internally, not supplied as additional target-view observations. Mask randomization reduces SSIM from 0.660 to 0.423 and increases LPIPS from 0.144 to 0.504, confirming that the model uses this spatial structure.
We do rely on an initial ego image to establish the target viewpoint and appearance. Acquiring one reference image generally requires less capture effort than continuously recording a complete first-person video. Where brief access to the target viewpoint is possible, one image can be captured at the start, while exocentric video supplies subsequent dynamics without continuous first-person recording.
The initial frame does not reveal subsequent actions, interaction changes, or newly visible regions; these still require prediction using exocentric information. Our repeated-first-frame baseline and residual-motion analysis examine this distinction. Any2Ego reaches residual correlation 0.647 and a residual-magnitude ratio of 1.04, supporting agreement with the actual continuation beyond retaining initial appearance.
Our method targets scenarios where exocentric video and one initial target-view image are available, reducing target-view RGB observations from a complete video to a single initial image.
Thank you again for your constructive assessment. We hope these additional results and clarifications address your concerns, and that you will consider raising your rating.