acceptodds
Under review as a conference paper at ICLR 2027

Any2Ego: Egocentric Viewpoint Translation Taking Textual Modality as Anchor

Abstract

Egocentric video is the format in which embodied agents see the world and is difficult to collect, whereas third-person recordings of the same activities are abundant. Translating these recordings requires recovering both the geometry and the appearance of interactions from the first-person viewpoint. Spatial priors constrain geometry, but leave appearance and scene details underspecified; using a text prompt alone does not determine which details supplement these priors or where they should influence generation. We introduce Any2Ego, which learns a first-person textual anchor complementary to hand–object interaction (HOI) masks and controls its influence through HOI-conditioned spatial routing. A Qwen3-VL-8B converter learns structured descriptions through supervised caption distillation with sentence-level weighting, followed by generation-grounded reinforcement that rewards agreement between added sentences and rendered previews. A lightweight gate then uses HOI features to modulate caption cross-attention at each video token. On held-out Ego-Exo4D clips, Any2Ego substantially improves perceptual fidelity over our reproduced EgoExo-Gen, better preserves viewpoint-dependent scene structure, and produces interactions whose appearance and temporal evolution more closely match the target egocentric videos.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Egocentric Video Generation, Spatial Conditioning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_5o5MOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Any2Ego, a framework for egocentric viewpoint translation from third-person videos. The method integrates predicted hand–object interaction (HOI) masks with a learned first-person textual anchor to jointly capture spatial constraints and semantic information. Specifically, a vision-language model is trained through supervised caption distillation and generation-grounded reinforcement learning to produce structured egocentric descriptions. These descriptions are subsequently incorporated into a video diffusion model through HOI-conditioned spatial routing, which adaptively modulates the contribution of textual information at different spatial locations.

Critical strengths
  • A meaningful integration of spatial and semantic conditioning. The paper addresses an important limitation of existing egocentric viewpoint translation methods by complementing HOI-based spatial constraints with learned first-person semantic descriptions. While HOI masks provide structural guidance for hand–object interactions, they cannot fully characterize viewpoint-dependent appearance, scene layout, and contextual details. The proposed textual anchor provides complementary semantic information, while HOI-conditioned spatial routing offers a reasonable mechanism for adaptively controlling its influence across different video regions. This integration is well motivated and practically relevant to egocentric video generation.
  • Convincing empirical improvements supported by controlled ablation studies. Any2Ego consistently outperforms the reproduced EgoExo-Gen baseline across multiple evaluation metrics, demonstrating improvements in structural similarity, perceptual fidelity, and video distribution quality. In particular, the controlled comparison between global and spatial caption injection in Table 3 provides meaningful evidence for the effectiveness of HOI-conditioned routing. The additional analysis of residual motion fidelity further strengthens the experimental validation by demonstrating improvements in temporal consistency beyond frame-level appearance similarity. These results provide reasonable empirical support for the proposed framework.
Critical weaknesses
  • Limited methodological novelty and insufficient mechanistic justification. Although the proposed framework demonstrates promising empirical improvements, its principal components are largely based on established techniques, including vision-language model distillation, group relative policy optimization (GRPO), cross-attention conditioning, and spatially adaptive gating. The main contribution therefore appears to lie in the integration and adaptation of existing mechanisms rather than the introduction of a fundamentally new modeling principle. While such integration can be valuable, the paper does not sufficiently articulate the distinctive methodological insight that makes the proposed approach more than a combination of complementary conditioning strategies. Furthermore, the theoretical analysis of spatial routing primarily characterizes the limitations of global response scaling under simplified assumptions. Although this analysis provides an intuitive motivation for spatial modulation, it does not directly explain how the learned HOI-conditioned gates achieve effective semantic localization or why they should outperform alternative spatial conditioning mechanisms.
  • Limited evidence of generalization and substantial dependence on strong initial conditions. The experimental evaluation is restricted to short cooking videos from Ego-Exo4D, with each generated sequence containing only 13 frames. Although this setting provides a controlled benchmark for egocentric viewpoint translation, it does not adequately establish the applicability of the proposed framework to longer video sequences, more diverse activities, or substantially different interaction scenarios. More importantly, Any2Ego requires the ground-truth egocentric first frame and its corresponding HOI mask as input. These conditions already provide substantial information about the target viewpoint, scene appearance, and initial interaction geometry, potentially reducing the difficulty of the viewpoint translation task. While this first-frame-conditioned formulation is explicitly defined in the paper and follows the adopted evaluation protocol, it remains unclear how effectively the proposed framework would operate when such strong target-view supervision is unavailable or imperfect.
Questions for authors
  • Could the authors further clarify the methodological distinction between the proposed HOI-conditioned spatial routing and existing spatially adaptive conditioning mechanisms? In particular, what evidence demonstrates that the learned routing gates provide meaningful semantic localization beyond conventional spatial feature modulation?
  • How does Any2Ego generalize to longer video sequences or activities beyond cooking? Additionally, how sensitive is the framework to the availability and accuracy of the initial egocentric frame and HOI mask? An evaluation under imperfect or reduced initial-view conditioning would help clarify its practical applicability.
1 comment
AuthorsOpenReview ↗

Response to Reviewer 5o5M: Spatial Routing, Generalization, and Input Conditions

Thank you for recognizing the motivation for combining spatial and semantic conditions and the value of our controlled ablations and residual-motion analysis. We clarify the contribution, generalization evidence, and practical input requirements below.

Q1: What distinguishes HOI-conditioned spatial routing, and what evidence supports semantic localization?

Our design connects two complementary conditions for first-frame-conditioned exo-to-ego generation: HOI prediction supplies interaction structure, while a learned first-person caption supplies appearance and scene information absent from structural labels. Spatial routing uses spatial-conditioning features to regulate the caption contribution at each video token.

Caption attention uses video queries and separate caption keys and values. A per-token sigmoid gate reads detached VACE conditioning features and, together with a learned per-block gain, scales the caption-attention response. This response is combined with native narration attention before the shared output projection. The trained HOI-conditioned base remains frozen, retaining the existing spatial and narration pathways (Eq. 6; Appendix I.5).

The matched Global-Spatial comparison provides direct evidence. Both use the same frozen base, captions, caption-projection architecture, and training schedule; Spatial adds one projection vector and one bias per main DiT block. Spatial improves FVD from 37.8 to 33.3, SSIM from 0.636 to 0.660, and LPIPS from 0.163 to 0.144 (Table 3). Residual correlation improves from 0.614 to 0.647 and residual PSNR from 20.47 to 21.17 dB (Table 7). These results support location-dependent caption modulation, which we interpret as interaction-conditioned spatial modulation rather than explicit word-to-region alignment.

Q2: What evidence supports generalization beyond cooking?

We added an evaluation on 80 fixed non-cooking ENIGMA-360 validation clips from 29 recordings, covering industrial activities. All methods use the same clips and metric implementation, without ENIGMA-360 fine-tuning (Appendix H, Table 8).

MethodPSNR ↑SSIM ↑LPIPS ↓FVD ↓
CogVideoX-5B V2V10.9240.37110.72312264.6
Wan2.1-VACE-1.3B10.5180.30610.71821742.5
LTX-Video11.4000.34350.67311518.4
Any2Ego Spatial12.6790.43320.49241022.8

All configurations use exo video or exo-derived conditions; all except CogVideoX also receive the initial ego frame. Training histories differ, so this evaluates the specified configurations rather than architecture alone under matched training.

Any2Ego achieves the best mean on all four metrics, supporting transfer from cooking to industrial activities without target-dataset fine-tuning. FVD comparisons are within this common 80-clip evaluation; absolute values should not be directly compared with the 1,000-clip Ego-Exo4D results because sample count and video distribution differ.

Q3: Does dependence on the initial ego frame and HOI conditions limit practical applicability?

We agree that these conditions provide important information. However, the subsequent ego HOI sequence is predicted from the exocentric video and initial conditions, without collecting or annotating the complete target ego video. Exocentric video is already an input to the task: subsequent HOI conditions are generated internally, not supplied as additional target-view observations. Mask randomization reduces SSIM from 0.660 to 0.423 and increases LPIPS from 0.144 to 0.504, confirming that the model uses this spatial structure.

We do rely on an initial ego image to establish the target viewpoint and appearance. Acquiring one reference image generally requires less capture effort than continuously recording a complete first-person video. Where brief access to the target viewpoint is possible, one image can be captured at the start, while exocentric video supplies subsequent dynamics without continuous first-person recording.

The initial frame does not reveal subsequent actions, interaction changes, or newly visible regions; these still require prediction using exocentric information. Our repeated-first-frame baseline and residual-motion analysis examine this distinction. Any2Ego reaches residual correlation 0.647 and a residual-magnitude ratio of 1.04, supporting agreement with the actual continuation beyond retaining initial appearance.

Our method targets scenarios where exocentric video and one initial target-view image are available, reducing target-view RGB observations from a complete video to a single initial image.

Thank you again for your constructive assessment. We hope these additional results and clarifications address your concerns, and that you will consider raising your rating.

Reviewer_9FYwOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses exocentric-to-egocentric video generation on Ego-Exo4D: given an exocentric video, the first egocentric frame and a task narration, it generates the subsequent egocentric frames. Building on EgoExo-Gen's two-stage design (cross-view HOI mask prediction followed by mask-conditioned generation), it adds a Qwen3-VL-8B converter that produces a structured first-person caption, trained with sentence-weighted distillation and a GRPO stage rewarded by CLIP agreement with rendered previews. The paper reports gains over a reproduced EgoExo-Gen and several off-the-shelf video generators.

Critical strengths
  • The freeze-frame control is a useful observation: on short horizons, frame-level metrics reward a model that simply holds the first frame.
  • The comparison between HOI-only, Global and Spatial variants is controlled, using the same frozen base, the same captions and the same training schedule.
  • Code and an offline video gallery are provided.
Critical weaknesses
  • Limited task and architectural novelty. The task and setting are taken directly from EgoExo-Gen, and the mask predictor, data protocol and two-stage structure are inherited from it. The main additions, a VLM captioner and a per-token gate on a new caption branch (described as "only 30 projection vectors and biases"), are modest. The surrounding framing (complementary textual anchors, the least-squares motivation in Eq. 7–8) appears constructed to justify a small change.
  • Incomplete related work. Several published exo-to-ego generation works are neither cited nor compared: Exo2Ego (ECCV 2024), Exo2Ego-V (NeurIPS 2024) and 4Diff (ECCV 2024). EgoX is cited only as an arXiv preprint although it appeared at CVPR 2026. On the captioning side, EgoInstructor (CVPR 2024), which uses exocentric videos to caption egocentric clips, is also missing.
  • Unfair or incomplete baseline comparison. All six general-purpose generators are run zero-shot from released checkpoints and have never seen Ego-Exo4D, whereas the original EgoExo-Gen paper fine-tuned its baselines on Ego-Exo4D. The only trained competitor is EgoExo-Gen, reproduced on a different backbone with different frame counts and resolution.
  • Qualitative evidence is weak. The qualitative improvements come from five hand-picked clips out of 1,000, and they are confined to background and scene framing (windows, cabinets), not to the hand–object interaction the task targets. The per-clip gains in these examples (e.g., SSIM +0.17) are well above the pool average (+0.05). No failure cases or random samples are shown, and the captions behind each example are not displayed.
Questions for authors
  • How does the reproduced EgoExo-Gen relate to the original? Please explain the large gap from the numbers in the original paper.
  • Can you compare against Exo2Ego and Exo2Ego-V, or explain why they are not applicable to your input setting?
  • Can you show randomly sampled results, failure cases, and examples where the hand–object interaction itself improves?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 9FYw: Contributions, Comparisons, and Generalization

Thank you for the thoughtful feedback, which helped sharpen our contribution and strengthen the evidence.

Q1: Contribution beyond EgoExo-Gen.

A1: Building on EgoExo-Gen's HOI prediction, Any2Ego learns complementary target-view semantics and integrates them through spatial control: what to add beyond interaction labels, and where to apply it.

Our converter learns first-person descriptions through sentence-weighted supervision and generation-consistency feedback. The caption branch retains native narration. Gates from detached VACE conditioning features scale its attention response per token, which is added to native attention before the shared output projection (Appendix I.5).

Table 3 demonstrates the benefit: HOI-only/Global/Spatial achieve SSIM 0.608/0.636/0.660 and LPIPS 0.194/0.163/0.144. The matched Global–Spatial comparison shows the gain from spatial modulation. Appendix I details the architecture; the 30 vectors and biases belong to the lightweight spatial gate. Eqs. 7–8 motivate its location-dependent scaling.

Q2: Related work and EgoX citation.

A2: Thank you for these references and the correction. We added Exo2Ego, Exo2Ego-V, 4Diff and EgoInstructor, and updated EgoX to CVPR 2026.

Exo2Ego uses a hand-layout prior; Exo2Ego-V combines view-translation priors with appearance and temporal modeling; 4Diff and EgoX use geometric priors. Any2Ego learns semantics left unspecified by structural labels and injects them with HOI-conditioned modulation, adding scene, appearance and lighting cues to the interaction scaffold. EgoInstructor captions observed ego video using retrieved exo instructional videos; our converter predicts semantics for the ego continuation to be generated. Revised Related Work explains these distinctions.

Q3: Baseline training conditions.

A3: We evaluated 80 fixed non-cooking ENIGMA-360 validation clips without ENIGMA fine-tuning.

MethodPSNR↑SSIM↑LPIPS↓FVD↓
CogVideoX-5B V2V10.9240.37110.72312264.6
Wan2.1-VACE-1.3B10.5180.30610.71821742.5
LTX-Video11.4000.34350.67311518.4
Any2Ego Spatial12.6790.43320.49241022.8

All use exo video or exo-derived conditions; all except CogVideoX also use the ego first frame. Any2Ego leads on all four means. FVD is 32.6%/41.3%/54.8% lower than LTX/Wan/CogVideoX on these 80 clips (distribution-distance reductions, not perceptual-quality percentages). This supports cross-domain transfer; training histories remain different. Appendix H details the inputs and protocol.

Q4: Improvements mainly concern background and framing.

A4: Thank you for this observation. It reflects the intended division of roles in Any2Ego: HOI provides strong interaction-structure constraints, and the caption branch supplies complementary scene and appearance information.

Sentence-weighted SFT emphasizes information beyond what the initial ego frame, masks and narration already express. Spatial injection uses VACE conditioning features from the initial ego frame and predicted HOI, together with per-block gains, to regulate the caption-attention response. Together, these mechanisms are designed to preserve the interaction scaffold while enriching the generated scene.

Captions can therefore enrich hand/object texture and appearance while producing more visible gains in background layout, framing and lighting. This pattern matches the module's purpose: complementing structural control with richer visual context for coherent target-view generation. Appendix I explains this rationale.

Q5: Difference from the original EgoExo-Gen scores.

A5: Our implementation retains EgoExo-Gen's HOI conditioning principle and uses Wan2.1-VACE-1.3B as the shared generator; the original initializes from SEINE. We evaluate on our fixed 1,000-clip pool, generating 13 frames at 480×480 with image metrics computed at 256×256. The original reports 16-frame generation at 256×256 on its evaluation pool.

Backbone, samples and protocol differ. Our shared-backbone comparisons isolate the added branch and spatial modulation. We will clearly identify the adaptation and protocol.

Q6: Comparison with Exo2Ego and Exo2Ego-V.

A6: Thank you for this suggestion. Related Work now compares these methods with our learned semantics and HOI-guided injection. Our setting uses one exo video and an initial ego frame. Exo2Ego-V's default setup additionally uses multiple exo views and camera/pose conditions.

In the public releases inspected, we found Exo2Ego evaluation utilities and Exo2Ego-V's view-translation prior, but could not locate the complete Exo2Ego generator checkpoints or Exo2Ego-V's trained reference and denoising UNets needed for numerical comparison. We therefore provide the methodological comparison alongside the completed controlled experiments and external-domain evaluation.

Thank you. In light of these results and clarifications, we hope you will consider raising your rating.

Thanks for the authors' clarification on the relationship with other similar works. I would raise the score and strongly suggest the authors to clarify how their work compares with others in the final version. And, if possible, any form of qualitative comparison with more baseline methods even if the settings are different.

Dear Reviewer 9FYw,

Thank you for raising your score and for your constructive suggestions. In the final version, we will clarify how our method compares with related approaches, including differences in inputs, conditioning mechanisms, and evaluation settings. We will also aim to include qualitative comparisons with additional baselines, clearly noting any differences in their settings to help readers interpret the results.

Thank you again for recognizing our work and for your valuable suggestions to improve the paper.

Best regards,
The Authors

Reviewer_hsEuOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Any2Ego generates an egocentric video from an exocentric video, an initial ego frame and task narration. It predicts hand-object interaction masks and learns a first-person caption with six fields describing hands, objects, contact, scene, motion and lighting. The caption converter is trained through weighted distillation and GRPO with a rendered-preview reward. A separate caption-attention branch in Wan2.1-VACE uses HOI features to adjust the caption's spatial influence. On 1,000 held-out Ego-Exo4D cooking clips, the full method reaches FVD 33.3, compared with 42.6 for the reproduced HOI-only baseline and 37.8 for global caption injection. The evaluation also includes a repeated-first-frame baseline and residual-motion measurements.

Critical strengths
  1. Spatial routing improves the results in a reasonably controlled comparison. Global and Spatial share the HOI-conditioned base, captions, projection architecture and training schedule, while the added gate improves FVD from 37.8 to 33.3 and LPIPS from 0.163 to 0.144 (Table 3). Residual correlation also rises from 0.614 to 0.647, which helps connect the gain to the predicted continuation.

  2. The repeated-first-frame baseline is a particularly helpful part of the evaluation. It achieves 0.601 SSIM and 21.88 dB PSNR, ahead of several video generators despite producing no motion (Table 2). Including this result alongside FVD and residual-change measurements makes the short-horizon comparison much easier to interpret.

  3. The structured caption gives the generator information that the HOI masks cannot fully describe, especially background layout and appearance. The improvement from HOI-only to Global, together with the caption-randomization experiment, shows that the model uses this information. The cabinet, window and cooking-area examples make the role of the caption concrete.

Critical weaknesses
  1. The caption-training stages are evaluated mainly through an automatic caption judge. The paper does not compare videos generated from unweighted-SFT, weighted-SFT and GRPO captions under a fixed generator. It is therefore hard to tell how much the extra caption-training machinery contributes to the final result, beyond adding a useful caption at all.

  2. The trained native text pathway is a close baseline on frame-level metrics, reaching PSNR 22.195 and LPIPS 0.1511 versus 22.28 and 0.144 for the full model (Table 4). Its FVD and residual-motion results are missing. Since the paper shows that frame metrics can be misleading on these clips, this leaves an important gap in the case for a separate caption branch.

  3. The GRPO reward measures similarity between a candidate caption and a video generated from that same caption (Eq. 4). A description that is easy for the preview generator to render could receive a high score even when it misstates the actual interaction. The reward's relationship to factual cross-view accuracy needs closer examination, particularly for newly visible content and fine hand-object contact.

  4. Some details in Figure 2 are difficult to follow. Several arrow endpoints do not clearly identify the receiving module. In panel (c), the relationship between the ego first frame and the noisy latents is ambiguous: it is unclear whether they are combined before entering the subsequent transformer blocks or supplied through separate paths. The figure should show these inputs and their connections more clearly.

Questions for authors
  1. Using the same video generator and routing setup, what are the FVD and residual-motion results with unweighted-SFT, weighted-SFT and GRPO captions? How much downstream gain comes specifically from the GRPO stage?

  2. What are the FVD and residual-frame results for Native n, and how do its trainable parameters and training budget compare with the dedicated caption branch?

  3. Does a higher preview reward predict more accurate captions against the paired ego observations? Could you examine high-reward errors or provide a targeted assessment of contact descriptions and newly visible objects before and after GRPO?

  4. Could you clarify the arrow endpoints and input paths in Figure 2? In panel (c), how are the ego first frame and noisy latents related, and do they enter the subsequent transformer blocks together or through separate paths? Please make this distinction explicit in the diagram.

Comments on the paper

1 comment
Area_Chair_6kpmOpenReview ↗

A Gentle Nudge: Let's Keep the Discussion Going!

Dear Reviewers,

The authors have now posted their responses, addressing your concerns and incorporating your suggestions. We are already halfway through the discussion period!

One of the distinguishing features of ICLR is its interactive review process, where constructive discussions not only help strengthen individual papers but also contribute valuable insights to the broader scientific community.

So, let's not let the discussion be a one-way street! I encourage you to revisit the authors' responses, acknowledge their efforts, and engage in further discussion where needed. After all, a good scientific discussion is often as valuable as the final verdict!

-AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.