acceptodds
Under review as a conference paper at ICLR 2027

All-in-One Unified Egocentric World Simulator

15 views

Abstract

Egocentric and exocentric videos are two views of the same physical event: the first-person view captures intent-driven hand–object interaction, while the third-person view reveals the camera wearer's full-body motion and the surrounding scene. Most cross-view generators translate between the two in one predetermined direction, so different directions and temporal completions require separate models. In this paper, we introduce EgoU, a unified generative framework that places one egocentric and up to three exocentric views on a shared video canvas and expresses diverse generation tasks through conditioning masks and optional text prompts. Fine-tuned from a pretrained video diffusion transformer (DiT), EgoU supports exocentric-to-egocentric generation with or without an egocentric seed frame, egocentric-to-exocentric generation, temporal extension, temporal interpolation, text-driven egocentric synthesis, and joint cross-view generation within a single set of parameters. We incorporate view-type embeddings to differentiate perspectives, condition on scene geometry and the wearer's head trajectory through a dedicated cross-attention branch, and apply an auxiliary pose loss to supervise exocentric human motion. To train and evaluate this formulation, we build EgoExoVideo-600K, a corpus of 604,113 synchronized ego–exo clip pairs, and EgoU-Benchmark, which scores all tasks on the same held-out clips. To our knowledge, EgoU is the first single model to support all of these tasks through a shared interface, and it matches or surpasses task-specific baselines. EgoU is open-sourced anonymously at huggingface.co/EgoVideo/EgoU-5B.

Primary area:
generative models
Keywords:
Egocentric Vision, World Simulator

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_BSagOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

EgoU uses Wan2.2-TI2V-5B as its backbone, concatenating the latent variables of first-person and up to three third-person videos along the width, and unifying seven generation tasks through conditional masks. The model incorporates viewpoint type embeddings, VGGT-Ω scene features, head trajectories, and Plücker ray conditions, and provides training supervision with a frozen pose estimator. The authors evaluated the same set of model weights on a test set from 420 independent takes, using 604,113 synchronized video pairs and 294,706 multi-view groups.

Critical strengths

Using the same model to handle different viewpoints and observation positions, the task definition is relatively clear. VAE encoding and decoding are performed independently for each viewpoint, avoiding problems caused by processing across panel boundaries. Coverage of seven task categories demonstrates the method's practicality.

Data processing is relatively complete; the appendix explains distortion correction, camera coordinate transformation, wearer tracking, pose labeling, source weights, and take-level data partitioning. Only one set is selected for each take during testing, which helps reduce sample correlation caused by adjacent segments.

Experiments include multiple baselines, module ablation, and human evaluation. Table 1 shows that both FVD and Align achieved leading results, with the highest human overall mean for six task categories. The method's engineering implementation and data preparation have received sufficient investment, warranting further validation.

Critical weaknesses
  1. Anchor points for interpolation tasks may contain information about the frames to be generated.

Appendix B.2 (page 16) states that during inference, the video from frame 0 to the pixel block corresponding to the last anchor point is encoded first,

and then a specified latent frame is taken as the anchor point. Page 18 gives the interpolation anchor points as S={0,f−1}, F=37, f=10, therefore, according to this description, all 37 frames need to be encoded. B.1 also points out that temporal grouping is not equivalent to independent receptive fields, and causal VAEs can utilize earlier frames.

The final anchor point constructed in this way may contain information about the intermediate frames to be generated.

Furthermore, Figure 16 on page 37 describes the conditions as the first frame and the last four frames, which differs from the observational differences of single-frame interpolation. It is necessary to clarify which frames were actually used for each baseline, and whether the scoring only covers the same interval to be generated.

This is a key issue affecting the interpretation of interpolation results. It is recommended to fix the allowed first and last observation conditions, replace or shuffle invisible intermediate frames, and check whether the extracted anchor latent variables have changed; then use an encoding method that only depends on visible pixels, and re-evaluate under the same observation conditions. If the actual task is latent space completion after complete video encoding, the task definition should also be clearly defined. Currently, the impact of this issue on numerical results cannot be determined solely by textual descriptions.

  1. The trajectory, viewpoint, and prefix frame conditions obtained by each method are not fully explained.

Pages 7-8 explain that the main table uses the real first-person trajectory. This is a clearly given condition, but it makes the results mainly reflect the generation capability of the given trajectory, especially in Extension, where future head movements themselves are also information that needs to be predicted. The paper clearly states that EgoX and Exo2Ego-V obtain the same trajectory, but does not explain item by item how Wan et al. baselines use trajectories, complete input videos, and multiple external viewpoints.

EgoU can read up to three external viewpoints, and some baselines were originally set to a single viewpoint; Extension uses a nine-frame prefix, while I2V backbones are usually conditioned on the initial image. A single round of fine-tuning on the same data does not guarantee consistency of these conditions. Another detail that needs explanation is that Text2Ego is described as using the first frame and text, and B.4 states that single-panel tasks disable geometric branches, but the ray embedding in Equation (6) exists independently. Please explain whether single-panel tasks still use real trajectories or ray conditions. It is recommended to list the number of input viewpoints, observation frames, text, complete or prefix trajectories, and other geometric information by task and method, and report the performance of the same complete model with and without real trajectories. Retraining a model with the head pose branch removed in Table 2 answers the module contribution question; it cannot replace the test of the complete model lacking trajectories during inference.

  1. Ablation has not yet separated the benefits of unified training itself. Table 2 compares direct training, canvas, and geometry and pose modules, but lacks a comparison of single-task canvas models with the same backbone, training data, input conditions, and computational budget. Therefore, it is currently difficult to distinguish whether the improvement comes from new conditions, structural changes, or transfer brought about by multi-task joint training. This point is also related to contribution localization. EgoX already has conditional stitching in the width direction [1], VACE unifies multiple conditional and masking tasks [2], and Syn2Seq-Forcing involves bidirectional view generation [3]. The valuable difference in this paper lies in the combination of simultaneous multi-view, unified seven-task and geometric design. It needs to be demonstrated which single-task solutions bring benefits that cannot be obtained by this combination, and whether the performance of some tasks is sacrificed. It is recommended to prioritize increasing the single-task and joint training comparison of budget matching, and then ablate view type embedding and temporal bias respectively, and report the quality, latency and memory overhead under different number of views.

  2. The statistical support for consistency evaluation and human evaluation is weak. FVD and image metrics are reported on 420 groups, but Align only covers 20 groups, including 47 first-person problems and 34 third-person problems. Appendix E (pages 24-25) also states that only 10 sets of human evaluations were used for Extension and Interpolation; this exception should be explained in the main text regarding the 20 sets of evaluation samples.

Table 1 does not provide confidence intervals for the differences. Text2Ego's Overall score is 3.68 vs. 3.66, making it difficult to confirm a stable improvement in human preference; Ego2Exo's 3.71 is also lower than A14B's 3.78. The overall trend is positive, but the reliability of all differences cannot be judged solely by the mean.

Align asks questions separately for each target perspective and then averages them, primarily reflecting semantic event matching, but not directly examining whether the actions, contact relationships, and spatial layouts of multiple generated perspectives are consistent at the same time. The lack of video filtering and time reversal comparisons is helpful, but the problem selection relies on the same evaluation model and is only called once; its stability still needs evaluation. The correlation between cross-method and cross-task means cannot replace sample-level validation.

It is recommended to expand the independent take samples, provide paired confidence intervals and task-specific correlations, and add direct cross-view synchronous evaluation.

Pixel and feature metrics should specify whether conditional frames were excluded; when generating multiple external views, it should be clearly stated whether all views are evaluated or only one view is selected.

Questions for authors
  1. Are the interpolation anchor points obtained from the complete video encoding containing unknown intermediate frames? After fixing visible frames and replacing intermediate frames, do the anchor points remain unchanged?

  2. What specific trajectories, viewpoints, and observation frames are obtained for each baseline? How does the same complete model perform without inputting real trajectories?

  3. Under the same data, conditions, and training budget, what are the gains or losses of the joint model compared to the single-task model?

  4. After expanding the sample and providing intervals, do the advantages of Align and human evaluation still hold? What are the results of direct cross-viewpoint simultaneous evaluation?

8 comments
AuthorsOpenReview ↗

Comment 1/6 (W1, Q1: the interpolation anchors)

Thank you for the detailed and constructive review. It identified an error in how we extracted the interpolation anchors, which is fixed in this comment, and your other suggestions led to the single-task control, the component ablations, the trajectory-free test, the cost measurements and the enlarged evaluation in the comments that follow.

W1, Q1. The interpolation anchors are taken from an encode of the full clip with a causal VAE, so the last anchor may contain information about the intermediate frames to be generated. Fix the visible frames, replace or shuffle the hidden ones and check whether the anchors change; then re-evaluate with an encoding that depends only on visible pixels. Figure 16 describes the condition as the first frame and the last four; clarify which frames each baseline uses and whether scoring covers the same interval.

Your diagnosis is correct: the anchor latents as originally extracted did carry information about the hidden frames. We reproduced your reasoning, quantified the effect, fixed it and re-evaluated. The corrected Interpolation row is 22.13 dB PSNR on all 420 groups, and EgoU remains ahead of the three baselines below on all four metrics.

The mechanism. The Wan VAE is causal, so latent index 9 has a receptive field reaching back over pixel frames 1–32, exactly the span the interpolation task withholds. Encoding the clip up to the last anchor and slicing latent 9, as Appendix B.2 described, therefore produced an anchor that was a function of frames the model was supposed to predict. This is a property of the encoder's receptive field; it is not affected by the VAE being frozen.

How large it is. We ran the probe you propose on the released checkpoint: fix the visible frames (0 and 33–36), replace the hidden frames 1–32 either with a shuffled copy or with a constant gray, then re-extract the anchors and re-run interpolation (50 groups, all else identical). Had the anchors been independent of the hidden frames, the three arms would coincide. Before the fix they differ by 0.70 dB under shuffling and 0.83 dB under gray.

The fix. The sparse-anchor path now encodes only the pixel frames that the task declares visible. For S = {0, f−1} it encodes 5 frames (0, 33, 34, 35, 36) instead of 37. A unit test asserts, over 13 anchor patterns, that every frame handed to the encoder belongs to the declared visible set.

Verification. After the fix, the same three-arm probe on the same 50 groups returns the same value in all three arms: the anchors no longer depend on the hidden frames.

Hidden framesPSNR before the fixPSNR after the fix
unchanged22.80822.125
shuffled22.10622.125
gray21.97622.125

After the fix the three arms are identical to four decimal places on every one of the 50 groups and at every canvas width, on SSIM and LPIPS as well as PSNR. On these 50 groups the fix lowers PSNR from 22.808 to 22.125, a drop of 0.68 dB.

What changes in the paper. The correction affects only patterns whose observed latent indices are non-contiguous, which among the seven families is interpolation alone. For all, first and prefix_k, a latent prefix of the full encode is by causality exactly the encode of the corresponding pixel prefix, so those rows are bit-identical before and after the fix. Table 1 therefore still reports a single checkpoint, and only the Interpolation row moves. The corrected row, measured on all 420 groups from the released checkpoint, is:

Interpolation (VI2V)FVDPSNRSSIMLPIPS
Wan2.1-14B FLF2V204.219.380.6730.305
Framer225.320.320.6820.311
SEINE425.719.910.6710.313
EgoU (corrected)159.222.130.7260.241

The baselines in this table are re-measured through the same pipeline of encoding 5 frames (0, 33, 34, 35, 36), so all rows are measured under the same protocol.

Figure 16. Under the causal grouping, latent frame 9 corresponds to pixel frames 33–36, so "the first frame and the last four" is the pixel-level description of S = {0, f−1}: five visible pixel frames, entering the model as one anchor latent at each end.

Scoring interval. Scoring covers only the generated interval: conditioned frames are excluded from every pixel and feature metric, for our model and for every baseline, and all methods receive the same visible frames.

Revision. We will correct the anchor extraction in Appendix B.2, replace the Interpolation row of Table 1 with the corrected row, and make the latent-to-pixel correspondence explicit in the caption of Figure 16. At the meanwhile, all interpolation results reported in the rebuttal, including human study and re-calculation of Align score, use videos regenerated with the corrected visible-only anchor extraction.

AuthorsOpenReview ↗

Comment 2/6 (W2, Q2: conditioning and trajectory-free inference)

W2. The main table uses the real ego trajectory, which in Extension includes future head motion, and the paper does not explain item by item which trajectories, input views and observed frames each baseline receives. EgoU reads up to three exo views while some baselines use one; Extension uses a nine-frame prefix; B.4 says single-panel tasks disable the geometry branch, yet the ray embedding of Eq. (6) exists independently.

The conditions each method receives are listed below. At a matched input count, EgoU remains 0.65 dB above A14B on Exo2Ego.

Trajectories. EgoU, EgoX, and Exo2Ego-V receive ground-truth head trajectories. The two adapted Wan2.2 baselines also receive head trajectories through cross-attention. WorldWander, EgoExo-Gen, Framer, SEINE and Wan2.1-14B FLF2V do not receive a trajectory. In Extension, a full trajectory supplies future head motion, so the corresponding result measures generation conditioned on that motion.

View counts. The audited A14B multi-view runs use two-panel canvases, while EgoU uses up to four panels. Additional exo inputs improve EgoU's Exo2Ego PSNR by 0.60 dB from one to three input views on the 104-group matched subset. When those panels are generation targets in Ego2Exo, exo1 PSNR decreases by 0.44 dB. So the extra panels help EgoU on Exo2Ego and cost it on Ego2Exo relative to a two-panel run. At a matched input count the Exo2Ego advantage remains: in the same unseeded V2V setting, with a single exo view and on the same 104 groups, EgoU at N=2 reaches 15.02 dB against 14.37 dB for A14B, a margin of 0.65 dB, so the PSNR advantage over A14B does not depend on the additional views.

Method, Exo2Ego (V2V)Panels Nexo views givenPSNR
Wan2.2-A14B2114.37
EgoU2115.02
EgoU3215.36
EgoU4315.62

Extension prefix. EgoU and both adapted Wan baselines receive the first nine pixel frames of each panel, corresponding to three latent frames.

Single-panel conditioning. In the evaluated N=1 path, geometry cross-attention, ground-truth trajectory conditioning, and Plücker ray embeddings are active. The reported Text2Ego result therefore measures generation from text, a seed image, and trajectory conditioning.

Q2. How does the same complete model perform without real trajectories as input?

We evaluated Text2Ego and Extension with the trajectory supplied or withheld at inference using the same complete checkpoint. The unavailable-trajectory mode uses the all-zero track and availability flag used by training-time trajectory dropout (probability 0.4).

TaskConditionPSNR ↑SSIM ↑LPIPS ↓
Text2EgoTrajectory given14.510.5240.649
Text2EgoTrajectory withheld14.440.5310.677
ExtensionTrajectory given21.360.7330.243
ExtensionTrajectory withheld21.080.7230.266

For Text2Ego, withholding the trajectory reduces PSNR by 0.07 dB and increases LPIPS by 0.028, while SSIM increases by 0.007. For Extension, withholding the trajectory reduces PSNR by 0.28 dB and SSIM by 0.010, and increases LPIPS by 0.023. The trajectory improves all three metrics for Extension and gives mixed results for Text2Ego.

Revision. We will document the observation interface separately for each baseline and task, correct Appendix B.4 and the Figure 2 label, update the Text2Ego description, including Appendix D, to state its inputs, and add the matched-count and trajectory-free tables.

AuthorsOpenReview ↗

Comment 3/6 (W3, Q3: isolating unified training, component ablations, prior work)

W3, Q3. The ablation does not separate the benefit of unified training itself: there is no single-task canvas model with the same backbone, data, conditions and budget, so it is unclear what the joint model gains or loses, and whether some tasks are sacrificed. EgoX, VACE and Syn2Seq-Forcing each cover part of the design. Then ablate the view-type embedding and the temporal bias.

We ran the budget-matched single-task control and both ablations. Under identical backbone, data, conditioning and 10,000 steps, joint training costs 0.19 dB on Ego2Exo and gains 0.42 dB on Exo2Ego.

Single-task control. We trained two further canvas models that differ from the "+ Canvas Training" row of Table 2 in a single argument, the task-sampling distribution: one sees only Ego2Exo, the other only Exo2Ego. Everything else is held fixed: the same Wan2.2-TI2V-5B initialization, the same EgoExoVideo-600K training split, the same conditioning patterns and panel-count distribution, four GPUs with gradient accumulation 2, and the same 10,000-step budget. Both are evaluated on all 420 groups under the protocol of Table 2.

Model (10,000 steps, canvas, no geometry, no pose)Ego2Exo PSNR / SSIM / LPIPSExo2Ego PSNR / SSIM / LPIPS
Single-task (new control)23.78 / 0.840 / 0.09213.87 / 0.491 / 0.637
Multi-task (Table 2, + Canvas Training)23.59 / 0.836 / 0.09514.29 / 0.491 / 0.614

Sharing one model costs Ego2Exo 0.19 dB and gains Exo2Ego 0.42 dB and 0.023 LPIPS (SSIM is unchanged at 0.491); the direction that gains is the one in which no pixel of the target view is given. With geometry and pose conditioning added, the complete model (Table 2, last row) is 0.72 dB and 1.48 dB above the single-task models on the two directions.

Component ablations. We trained two arms at the same 10,000-step budget, changing one thing each: one with the view-type embedding removed, one with the temporal bias on the geometry cross-attention removed. Everything else (backbone, data, conditioning, schedule, seed) is identical. All three models are scored on the same 420 groups.

ConfigurationEgo2Exo PSNR / SSIM / LPIPSExo2Ego PSNR / SSIM / LPIPS
complete model24.50 / 0.850 / 0.08115.35 / 0.528 / 0.567
− view-type embedding24.28 / 0.842 / 0.08714.68 / 0.511 / 0.592
− temporal bias (geometry)24.36 / 0.844 / 0.08615.00 / 0.509 / 0.602

Removing either component lowers every metric in both directions. Without the view-type embedding the model loses 0.22 dB PSNR, 0.008 SSIM and 0.006 LPIPS on Ego2Exo, and 0.67 dB, 0.017 SSIM and 0.025 LPIPS on Exo2Ego. Without the temporal bias on the geometry cross-attention it loses 0.14 dB, 0.006 SSIM and 0.005 LPIPS on Ego2Exo, and 0.35 dB, 0.019 SSIM and 0.035 LPIPS on Exo2Ego.

Positioning relative to prior work. We agree the width-wise conditional stitching of EgoX, the unified conditioning and masking of VACE, and bidirectional view generation in Syn2Seq-Forcing each overlap one axis of this design. Our contribution is the combination (simultaneous multi-view panels, seven families under one conditioning interface, and geometry plus trajectory conditioning that is shared across all of them), together with the measurement of what it costs individual tasks in the single-task control above.

Revision. We will add the single-task control and both ablation rows to the paper, and word the positioning against EgoX, VACE and Syn2Seq-Forcing as above.

Reviewer_zyqVOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents a unified model for conditional egocentric-exocentric video generation tasks. The main highlight is by flexibly conditioning ego and exo views, EgoU can achieve seven ego-exo video gen tasks in a single model. The paper also utilized several technical designs such as 3D scene geometry information, camera tokens, camera rays, etc to better encode additional information into the ego-exo pipeline. The paper combines three datasets to train their model and demonstrate superior performance than previous approaches. The experimentation, ablation study, and user study are comprehensive.

Critical strengths

The main strength of this paper is the new pipeline for unifying seven ego-exo video gen tasks into a single model, with a set of designs on encoding the 3D scene, camera pose, and camera ray information. This is the first work to unify ego and exo views generation, compared to previous works that focus on a single specific ego-exo task. The design components adopted in the pipeline also look reasonable for the ego-exo task. This research direction is meaningful and has great potential for skill learning in robotics.

Critical weaknesses

The major weakness is that although this work achieve better performance than other methods, its qualitative performance still shows substantial discrepancies from the ground-truth, especially for future frames, such as (t=24, 36) in Fig. 4 of main paper. As a result, it remains unclear for the practical applicability of the proposed method in real-world use cases. This is particularly important for the content involving hand–object interactions, where inaccurate or misaligned ego/exo generation could lead to failures in downstream tasks such as robot manipulation.

Questions for authors
  1. For the three datasets used in the experiments, do the authors perform any data filtering, particularly to remove segments without meaningful hand–object interactions?
  2. For Eq. (7) in the main paper, what is the motivation for the β\beta subtracted term? And is this part ("Unified Temporally Biased Cross-attention") ablated in the experiments?
  3. Since this is a unified model supporting various number of exo views and ego view, I am wondering if the authors find any insights on how many exo views are required to generate meaningful ego views?
  4. What are the most challenging part to generate in ego/exo videos and how to tackle them in the future work?
2 comments
AuthorsOpenReview ↗

Comment 1/2 (W1, Q1, Q2: late-frame gap, data filtering, Eq. (7))

Thank you for the encouraging review and for pointing to robot skill learning as a use of this work. Your questions led to two additions reported below: an ablation of the temporal bias in Eq. (7) and a study of how many exo views are needed.

W1. Although EgoU outperforms other methods, its results still differ substantially from the ground truth, especially for later frames (t = 24, 36 in Figure 4), so its practical applicability is unclear, particularly for hand-object interaction and downstream robot manipulation.

We agree: a gap to ground truth remains at later frames, and hand-object contact is where it matters most for robot manipulation. EgoU is a 5B model trained within one epoch on a corpus that is small by the standards of video generation, and we expect both larger ego-exo data and a larger backbone to narrow the gap. The content that is hardest to generate is described under Q4.

Q1. Do the authors perform any data filtering on the three datasets, particularly to remove segments without meaningful hand-object interactions?

Yes, at the atomic-action stage: segments without a described action are dropped. There is no filter specific to hand-object interaction. Hand-object interaction is concentrated in the Ego-Exo4D portion. We keep segments that contain none, for example the walking and locomotion sequences from Nymeria and EgoExo-Synthetic, because they supply large, clean camera motion, which we expect to help the cross-view alignment that the geometry and ray conditioning must learn. Removing them would shift the corpus towards stationary manipulation.

Q2. What is the motivation for the subtracted term in Eq. (7)? Is this part ("Unified Temporally Biased Cross-attention") ablated?

The subtracted term −β_src(n) (u_i − τ_n)² is a quadratic temporal-distance penalty on the cross-attention logits: it attenuates attention from a generated position to geometry or camera tokens that are temporally distant from it. Without it, the logits carry no explicit preference for temporally nearby geometry, and the branch can behave like a global scene prior rather than a time-aligned one. β is per source so that the geometry tokens and the camera tokens can decay at different rates, and the penalty is quadratic so the falloff is smooth and controlled by a single rate β per source.

We ran this ablation for the geometry source, at the same 10,000-step budget as the rows of Table 2 and on the same 420 groups. Everything else (backbone, data, conditioning, schedule, seed) is identical. The table also includes the arm without the view-type embedding.

ConfigurationEgo2Exo PSNR / SSIM / LPIPSExo2Ego PSNR / SSIM / LPIPS
complete model24.50 / 0.850 / 0.08115.35 / 0.528 / 0.567
− view-type embedding24.28 / 0.842 / 0.08714.68 / 0.511 / 0.592
− temporal bias (geometry)24.36 / 0.844 / 0.08615.00 / 0.509 / 0.602

Removing the bias from the geometry source lowers every metric in both directions. On Ego2Exo it costs 0.14 dB PSNR, 0.006 SSIM and 0.005 LPIPS; on Exo2Ego it costs 0.35 dB, 0.019 SSIM and 0.035 LPIPS. The effect is present in both directions and is larger on Exo2Ego.

Revision. We will state the late-frame gap in the limitations section and add the temporal-bias row to Table 2.

AuthorsOpenReview ↗

Comment 2/2 (Q3, Q4: number of exo views, hardest part and next steps)

Q3. Are there any insights on how many exo views are required to generate meaningful ego views?

The model runs with a single exocentric view, and each further view improves the ego result: +0.60 dB PSNR from one view to three. We measured this with one checkpoint, inference only, so no training difference enters the comparison, on the 104 groups that every arm could score at every panel count.

Exo2Ego (V2V), where the number of input exo views grows and the single generated and scored panel is the ego view:

Method, Exo2Ego (V2V)Panels Nexo views givenPSNR
Wan2.2-A14B2114.37
EgoU2115.02
EgoU3215.36
EgoU4315.62

Every additional exocentric view helps: the second is worth +0.34 dB and the third a further +0.26 dB, and the gain has not saturated at three views. At a matched input count of one exo view, EgoU is 0.65 dB above the Wan2.2-A14B baseline on the same 104 groups.

Ego2Exo (VI2V) measures something different: there the conditioning is the same ego panel in every arm and what grows is the number of exo panels the model must generate jointly, while the scored panel is always exo1.

Panels Npanels generatedPSNRSSIMLPIPS
2124.6700.86580.0749
3224.4410.85690.0803
4324.2310.85620.0796

Each additional panel generated in the same canvas costs the scored panel about 0.22 dB PSNR; SSIM and LPIPS change little between three and four panels. Joint multi-panel generation therefore has a measurable cost on the scored panel.

Q4. What is the most challenging part to generate in ego/exo videos, and how can it be tackled in future work?

In the ego view, the hardest content is large scene change: when the wearer turns their body the view changes almost entirely, and heavy hand motion is hard to keep right at the same time. In the exo view, the hardest cases are a subject who moves extensively through the scene, and scenes with several people.

For future work we see two directions. For the ego view, supervision at the level of the hands, in the same way the current model supervises exo body pose. For both views, training data with more clips of large viewpoint change and of several people, which connects to the data-scale question under W1.

Revision. We will add both view-count tables to the paper.

Reviewer_LxH9OpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper deals seven egocentric and exocentric video tasks by one video diffusion model. So one model can do all tasks. It puts one ego view and maximum three exo views side by side to the shared latent canvas. And it uses mask to show which view and frame is given. During training it uses view-type embedding, cross-attention branch that gets scene geometry and head trajectory (optional), and add exo body pose loss for the Ego2Exo task. The authors made new dataset that is EgoExoVideo-600K dataset. However, they evaluate on 420 groups from Ego-Exo4D that is excepted in training. With one checkpoint it achieved positive score on FVD and Align in all seven tasks. Also PSNR is more higher than fine-tunned Wan2.2-5B.

Critical strengths
  1. I think that single checkpoint can does every jobs is the biggest advantage. Both-way translation, temporal extension, interpolation, text-based generation, joint generation are all possible. I think this is very practical contribution.
  2. The authors marked dataset and benchmark very detail. The authors seem to be very careful for designing the Align metric.
  3. The performance got much higher than the adapted 5B backbone. For example, Ego2Exo FVD drops from 381.9 to 101.5.
Critical weaknesses
  1. Table 2 shows effect of the canvas, geometry and pose loss. However, effect of training many jobs together and effect of canvas itself is not separated. Also task sampling is not same. Ego2Exo and Exo2Ego is 0.35 but interpolation is only 0.08. So it's unclear that how much each job gets or loses by sharing one model.
  2. It's also unclear that the evaluation is enough. Align only uses 20 groups, and human evaluation only uses 10 groups in extension and interpolation. Also benchmark only uses the clips where the geometry is complete and the pose is visible, and all clips are from Ego-Exo4D with static exo camera. So I couldn't find how much the difference between methods is stable in different clips.
  3. Figure 4 uses the groups where camera or body motion is large, and Appendix F uses the groups where the model got highest score compared to baseline. The paper also mentions this clearly. However, I couldn't find the example where EgoU fails or baseline looks better. For example, A14B achieved better score on Quality in Ego2Exo (3.76 vs 3.56).
Questions for authors
  1. I'm curious about the performance if we train canvas model with only one task, with same conditioning and training budget. If there is this comparison, it will be clear that how much each task gets or loses by sharing one model.
  2. Could you report the uncertainty of the Align and human evaluation score difference using paired clips please? Because some differences in Table 1 are very small. For example, in Ego2Exo Overall, EgoU is 3.71 and A14B is 3.78.
  3. I couldn't find how many steps were used for the Table 1 checkpoint. The baselines are fine-tuned for one epoch, and all rows in Table 2 are 10,000 steps. Also the first row of Table 2 in Ego2Exo is same with Wan2.2-5B row in Table 1 (19.53, 0.639, 0.191). Could you report the training steps of each model please?
  4. I couldn't find the row only for view-type embedding in Table 2. So I'm curious about how much the embedding helps. Is there result that is trained without the embedding?
4 comments
AuthorsOpenReview ↗

Comment 1/4 (W1, Q1: unified training against task-specific models)

Thank you for the careful review and for recognising the practical value of one checkpoint that serves all seven tasks. Your questions led to three additions reported below: a single-task control, paired intervals for the human evaluation and Align, and an ablation of the view-type embedding.

W1, Q1. Table 2 does not separate the effect of training the tasks together from the effect of the canvas itself, so it is unclear how much each task gains or loses by sharing one model. How does a canvas model trained on only one task perform, with the same conditioning and training budget?

We ran this control. At matched conditioning and a matched 10,000-step budget, sharing one model costs 0.19 dB on Ego2Exo and gains 0.42 dB on Exo2Ego.

Setup. We trained two further canvas models that differ from the "+ Canvas Training" row of Table 2 in a single argument, the task-sampling distribution: one sees only Ego2Exo, the other only Exo2Ego. Everything else is held fixed: the same Wan2.2-TI2V-5B initialization, the same EgoExoVideo-600K training split, the same conditioning patterns and panel-count distribution, four GPUs with gradient accumulation 2, and the same 10,000-step budget. Both are evaluated on all 420 groups under the protocol of Table 2.

Model (10,000 steps, canvas, no geometry, no pose)Ego2Exo PSNR / SSIM / LPIPSExo2Ego PSNR / SSIM / LPIPS
Single-task (new control)23.78 / 0.840 / 0.09213.87 / 0.491 / 0.637
Multi-task (Table 2, + Canvas Training)23.59 / 0.836 / 0.09514.29 / 0.491 / 0.614
  • Sharing one model costs Ego2Exo 0.19 dB and gains Exo2Ego 0.42 dB and 0.023 LPIPS (SSIM is unchanged at 0.491). The direction that gains is the one in which no pixel of the target view is given.
  • The complete model is above the single-task models on both directions. With geometry and pose conditioning added (Table 2, last row), it reaches 24.50 / 0.850 / 0.081 on Ego2Exo and 15.35 / 0.528 / 0.567 on Exo2Ego at the same budget, 0.72 dB and 1.48 dB above the single-task models.

W1, task sampling. Task sampling is not equal: Ego2Exo and Exo2Ego are sampled at 0.35, interpolation at only 0.08.

The mixture weights the two cross-view families highest because they are the ones whose conditioning pattern varies with every panel count and seed draw, while interpolation instantiates a single highly constrained pattern (two anchor latents). The control above gives the two ends of this choice for the cross-view families: probability 0.35 in the mixture against 1.0 alone.

Revision. We will add the single-task control to the paper and state the mixture weights.

AuthorsOpenReview ↗

Comment 2/4 (W2, W3: evaluation coverage and failure cases)

W2. Align uses only 20 groups and the human evaluation only 10 groups for Extension and Interpolation; the benchmark keeps only Ego-Exo4D clips with complete geometry, visible pose and static exo cameras. It is unclear how stable the differences between methods are across clips.

We enlarged the human evaluation. Extension and Interpolation are now rated on 20 groups, like the other families; the study has 90 raters, each of whom gives MOS ratings to 20 videos across the 7 tasks. To show how stable the differences are across clips, we report clip-level paired confidence intervals (Q2): 16 of the 19 Overall comparisons have intervals excluding zero, all in EgoU's favour. Align is now computed on 48 groups instead of 20 (Q2). The study also adds a cross-view consistency item on five of the seven families, in which raters see the generated ego and exo panels of the same canvas side by side and judge whether the action, the contact and the spatial layout agree; eight of its eleven paired intervals exclude zero, all in EgoU's favour.

Cross-view consistency, EgoU minus baseline, 95% paired clip-level bootstrap (10,000 resamples, seed 0), 718 judgements over five families:

FamilyBaselinepairsdiff95% CI
Ego2ExoWan2.2-5B20+1.33[+0.62, +1.96]
Ego2ExoWan2.2-A14B19+0.39[+0.04, +0.75]
Exo2Ego (VI2V)Wan2.2-5B18+1.38[+0.70, +2.05]
Exo2Ego (VI2V)Wan2.2-A14B18+0.87[+0.44, +1.30]
ExtensionWan2.2-5B18+1.29[+0.72, +1.89]
ExtensionWan2.2-A14B18+0.44[−0.05, +0.89]
InterpolationFramer20+0.45[−0.15, +1.01]
InterpolationSEINE20+1.02[+0.33, +1.64]
InterpolationWan2.2-5B20+0.93[+0.13, +1.63]
JointWan2.2-5B20+1.62[+1.18, +2.05]
JointWan2.2-A14B20+0.24[−0.22, +0.71]

The benchmark uses Ego-Exo4D because it is the only corpus we are aware of that instantiates all seven task families on the same clips: it is the one source with time-synchronized ego and multiple static exo cameras, per-take body pose, and complete scene geometry, which is what lets a single benchmark change only the per-panel conditioning and leave everything else fixed. Nymeria contributes training data but carries a moving observer camera and no multi-exo canvas, so it cannot instantiate Ego2Exo; EgoExo-Synthetic has no human annotation of the kind Align relies on.

W3. Figure 4 and Appendix F show groups selected for large motion and for the highest margin over the baseline; there is no example where EgoU fails or a baseline looks better. For example, A14B is rated higher on Quality in Ego2Exo (3.76 vs 3.56).

We agree, and we are adding a failure section selected the opposite way from Figure 4 and Appendix F: the groups with the largest deficit against the strongest baseline on each family, shown with the same four columns as Figure 4. Ego2Exo is one such case: Wan2.2-A14B is rated above EgoU on Quality there (3.76 vs 3.56), and the Ego2Exo groups in the section are selected against it.

Revision. We will describe the enlarged human study and the reason for building the benchmark on Ego-Exo4D in the main text, and add the failure section.

AuthorsOpenReview ↗

Comment 3/4 (Q2: uncertainty on Align and the human evaluation)

Q2. Report the uncertainty of the Align and human-evaluation score differences using paired clips; some differences in Table 1 are very small, for example Ego2Exo Overall, 3.71 for EgoU against 3.78 for A14B.

We computed clip-level paired intervals on the enlarged study. 16 of the 19 comparisons have intervals excluding zero, all in EgoU's favour, and no interval lies below zero.

Protocol. For each family and each baseline we take the per-clip mean rating of each method, pair them by clip, and bootstrap the mean paired difference over clips (10,000 resamples, seed 0). The intervals are computed on the enlarged study rather than on the ratings the paper reported.

Overall MOS, EgoU minus baseline, 95% paired bootstrap CI, all 20 groups per family:

FamilyBaselinepairsdiff95% CI
Ego2ExoWan2.2-5B20+1.22[+0.84, +1.60]
Ego2ExoWan2.2-A14B20+0.11[−0.31, +0.55]
Ego2ExoWorldWander20+0.42[+0.03, +0.82]
Exo2Ego (V2V)EgoX20+1.02[+0.47, +1.57]
Exo2Ego (V2V)Exo2Ego-V20+2.13[+1.70, +2.56]
Exo2Ego (V2V)WorldWander20+1.84[+1.55, +2.12]
Exo2Ego (VI2V)EgoExo-Gen20+1.35[+0.88, +1.82]
Exo2Ego (VI2V)Wan2.2-5B20+1.56[+0.93, +2.12]
Exo2Ego (VI2V)Wan2.2-A14B20+0.57[+0.18, +0.91]
ExtensionWan2.2-5B20+1.20[+0.72, +1.67]
ExtensionWan2.2-A14B20+0.49[+0.23, +0.85]
InterpolationWan2.1-14B FLF2V20+0.74[+0.40, +1.16]
InterpolationFramer20+0.81[+0.35, +1.24]
InterpolationSEINE20+1.16[+0.61, +1.62]
InterpolationWan2.2-5B20+1.00[+0.43, +1.54]
JointWan2.2-5B20+0.81[+0.35, +1.28]
JointWan2.2-A14B20+0.19[−0.19, +0.55]
Text2EgoWan2.2-5B20+1.20[+0.81, +1.58]
Text2EgoWan2.2-A14B20−0.03[−0.46, +0.36]
  • The example you cite. Ego2Exo Overall against A14B is +0.11 [−0.31, +0.55] on the enlarged study (3.71 vs 3.78 in the submitted Table 1): EgoU is comparable to A14B.
  • Against the A14B baselines, each of which is a separate model fine-tuned for one task and about three times our size, the single EgoU model is ahead on Exo2Ego (VI2V) and Extension and comparable on Ego2Exo, Joint and Text2Ego, where the lower bounds are −0.31, −0.19 and −0.46 MOS.
  • Against every other baseline EgoU leads on every family.
  • Coverage. The enlargement doubled the ratings: Extension and Interpolation were previously compared on ten clip pairs and are now on twenty, so every row above rests on twenty groups.

Align. We enlarged the Align evaluation from 20 to 48 groups, with more questions per group, and compared EgoU with the strongest baseline of each family, paired by group.

Align, EgoU minus baseline:

FamilyBaselineframes scoredgroupsdiff95% CIperm. p
Exo2Ego (V2V)EgoXall 3748+0.110[−0.039, +0.254]0.209
Exo2Ego (VI2V)Wan2.2-A14B1–3648+0.044[+0.000, +0.111]0.500
Ego2Exo (VI2V)Wan2.2-A14B1–3648+0.028[−0.125, +0.168]0.598
JointWan2.2-A14B1–3648+0.026[−0.150, +0.221]0.876
Text2EgoWan2.2-A14B1–3648+0.122[+0.015, +0.236]0.064
ExtensionWan2.2-A14B9–3648+0.024[−0.228, +0.250]0.890
InterpolationFramer1–3248+0.043[−0.062, +0.171]0.751

EgoU's Align is above the strongest baseline on all seven families, by +0.024 to +0.122, and the interval excludes zero on Text2Ego.

Revision. We will replace the means in the human-evaluation table with these paired intervals, report Align on the 48 groups with the intervals above, word the comparison with A14B as above, and state the per-task training of the A14B baselines in the caption of Table 1.

AuthorsOpenReview ↗

Comment 4/4 (Q3, Q4: training steps and the view-type embedding)

Q3. How many steps were used for the Table 1 checkpoint? The baselines are fine-tuned for one epoch and all rows in Table 2 use 10,000 steps. Also, the first row of Table 2 on Ego2Exo is the same as the Wan2.2-5B row in Table 1 (19.53, 0.639, 0.191).

Step counts are per-process data-loader steps; one epoch is 65,249 steps. Every model is the best checkpoint on the held-out split within its budget.

  • Table 1, EgoU: one epoch; the best checkpoint is at step 56,000.
  • Table 1, baselines: one epoch.
  • Table 2, every row: 10,000 steps.

Table 1's Wan2.2-5B TI2V row and Table 2's first row are the same checkpoint: its best held-out score falls within the first 10,000 steps.

Q4. Table 2 has no row for the view-type embedding alone. How much does the embedding help? Is there a result trained without it?

It helps, by about 0.2 dB on Ego2Exo and about 0.7 dB on Exo2Ego at this budget. We trained two arms at the same 10,000-step budget as the other rows of Table 2, changing one thing each: one with the view-type embedding removed, one with the temporal bias on the geometry cross-attention removed. Everything else (backbone, data, conditioning, schedule, seed) is identical. All three models are scored on the same 420 groups.

ConfigurationEgo2Exo PSNR / SSIM / LPIPSExo2Ego PSNR / SSIM / LPIPS
complete model24.50 / 0.850 / 0.08115.35 / 0.528 / 0.567
− view-type embedding24.28 / 0.842 / 0.08714.68 / 0.511 / 0.592
− temporal bias (geometry)24.36 / 0.844 / 0.08615.00 / 0.509 / 0.602

Removing either component lowers every metric in both directions. Without the view-type embedding the model loses 0.22 dB PSNR, 0.008 SSIM and 0.006 LPIPS on Ego2Exo, and 0.67 dB, 0.017 SSIM and 0.025 LPIPS on Exo2Ego. Without the temporal bias on the geometry cross-attention it loses 0.14 dB, 0.006 SSIM and 0.005 LPIPS on Ego2Exo, and 0.35 dB, 0.019 SSIM and 0.035 LPIPS on Exo2Ego.

Revision. We will add both rows to Table 2 and state the shared checkpoint in its caption.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Author Final Remark

We sincerely thank all reviewers and the Area Chair for the time and care invested in evaluating our submission. We appreciate the recognition of EgoU's key strengths: a single checkpoint that serves seven ego–exo video tasks, the carefully documented EgoExoVideo-600K dataset and EgoU-Benchmark, the geometry and trajectory conditioning design, and the potential of this direction for robot skill learning.

Our responses were posted later in the discussion period than we would have liked. The reviews asked for stronger statistical support for the human evaluation, and recruiting 90 raters and collecting the enlarged study took most of the period. We apologise for the limited time this leaves for discussion.

What we added during the rebuttal

  • Interpolation anchors (BSag). The anchor extraction is corrected to encode visible frames only, verified with the probe the reviewer proposed, and EgoU and all interpolation baselines are re-evaluated under one protocol.
  • Unified versus single-task training (LxH9, BSag). A budget-matched single-task control: sharing one model costs 0.19 dB on Ego2Exo and gains 0.42 dB on Exo2Ego.
  • Component ablations (LxH9, zyqV, BSag). Models trained without the view-type embedding and without the temporal bias of Eq. (7).
  • Enlarged human study (LxH9, BSag). 20 groups for every family, clip-level paired confidence intervals, and a new item that judges the ego and exo views of one canvas jointly.
  • Align (LxH9, BSag). 48 groups, paired intervals against the strongest baseline of each family, and three judge calls per question.
  • Conditioning and cost (zyqV, BSag). The inputs each method receives, trajectory-free inference with the same checkpoint, and quality, latency, and memory against the number of views.
  • Failure cases and limitations (LxH9, zyqV). Groups selected for the largest deficit against the strongest baseline, the late-frame gap, and the hardest content.
  • Clarifications. Training steps and checkpoint selection, scoring rules, data filtering, the choice of Ego-Exo4D for the benchmark, and positioning against EgoX, VACE, and Syn2Seq-Forcing.

What changed in the revised PDF

All revised text is in blue. The title and abstract are unchanged.

  • Table 1: corrected Interpolation rows; the user-study column now gives paired differences with 95% intervals.
  • Table 2: three new rows (single-task control, view-type embedding, temporal bias).
  • Section 4: evaluation protocol, human evaluation, unified versus single-task training, and inputs at inference.
  • Appendix: corrected anchor encoding (B.2), Align and user-study intervals (D, E), and three new sections on failure cases, the observation interface of each method, and view-count, cost, and trajectory analyses.

Overall, we believe EgoU makes a timely contribution: it replaces a collection of direction-specific ego–exo generators with one model and one benchmark on which all seven tasks are scored on the same clips. The model weights are already public, and we will release the EgoExoVideo-600K annotations and EgoU-Benchmark. Thank you again for your thoughtful consideration; we would sincerely appreciate it if these responses and the revised paper could be taken into account during the AC–reviewer discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.