- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
EgoU uses Wan2.2-TI2V-5B as its backbone, concatenating the latent variables of first-person and up to three third-person videos along the width, and unifying seven generation tasks through conditional masks. The model incorporates viewpoint type embeddings, VGGT-Ω scene features, head trajectories, and Plücker ray conditions, and provides training supervision with a frozen pose estimator. The authors evaluated the same set of model weights on a test set from 420 independent takes, using 604,113 synchronized video pairs and 294,706 multi-view groups.
Critical strengths
Using the same model to handle different viewpoints and observation positions, the task definition is relatively clear. VAE encoding and decoding are performed independently for each viewpoint, avoiding problems caused by processing across panel boundaries. Coverage of seven task categories demonstrates the method's practicality.
Data processing is relatively complete; the appendix explains distortion correction, camera coordinate transformation, wearer tracking, pose labeling, source weights, and take-level data partitioning. Only one set is selected for each take during testing, which helps reduce sample correlation caused by adjacent segments.
Experiments include multiple baselines, module ablation, and human evaluation. Table 1 shows that both FVD and Align achieved leading results, with the highest human overall mean for six task categories. The method's engineering implementation and data preparation have received sufficient investment, warranting further validation.
Critical weaknesses
- Anchor points for interpolation tasks may contain information about the frames to be generated.
Appendix B.2 (page 16) states that during inference, the video from frame 0 to the pixel block corresponding to the last anchor point is encoded first,
and then a specified latent frame is taken as the anchor point. Page 18 gives the interpolation anchor points as S={0,f−1}, F=37, f=10, therefore, according to this description, all 37 frames need to be encoded. B.1 also points out that temporal grouping is not equivalent to independent receptive fields, and causal VAEs can utilize earlier frames.
The final anchor point constructed in this way may contain information about the intermediate frames to be generated.
Furthermore, Figure 16 on page 37 describes the conditions as the first frame and the last four frames, which differs from the observational differences of single-frame interpolation. It is necessary to clarify which frames were actually used for each baseline, and whether the scoring only covers the same interval to be generated.
This is a key issue affecting the interpretation of interpolation results. It is recommended to fix the allowed first and last observation conditions, replace or shuffle invisible intermediate frames, and check whether the extracted anchor latent variables have changed; then use an encoding method that only depends on visible pixels, and re-evaluate under the same observation conditions. If the actual task is latent space completion after complete video encoding, the task definition should also be clearly defined. Currently, the impact of this issue on numerical results cannot be determined solely by textual descriptions.
- The trajectory, viewpoint, and prefix frame conditions obtained by each method are not fully explained.
Pages 7-8 explain that the main table uses the real first-person trajectory. This is a clearly given condition, but it makes the results mainly reflect the generation capability of the given trajectory, especially in Extension, where future head movements themselves are also information that needs to be predicted. The paper clearly states that EgoX and Exo2Ego-V obtain the same trajectory, but does not explain item by item how Wan et al. baselines use trajectories, complete input videos, and multiple external viewpoints.
EgoU can read up to three external viewpoints, and some baselines were originally set to a single viewpoint; Extension uses a nine-frame prefix, while I2V backbones are usually conditioned on the initial image. A single round of fine-tuning on the same data does not guarantee consistency of these conditions. Another detail that needs explanation is that Text2Ego is described as using the first frame and text, and B.4 states that single-panel tasks disable geometric branches, but the ray embedding in Equation (6) exists independently. Please explain whether single-panel tasks still use real trajectories or ray conditions. It is recommended to list the number of input viewpoints, observation frames, text, complete or prefix trajectories, and other geometric information by task and method, and report the performance of the same complete model with and without real trajectories. Retraining a model with the head pose branch removed in Table 2 answers the module contribution question; it cannot replace the test of the complete model lacking trajectories during inference.
-
Ablation has not yet separated the benefits of unified training itself. Table 2 compares direct training, canvas, and geometry and pose modules, but lacks a comparison of single-task canvas models with the same backbone, training data, input conditions, and computational budget. Therefore, it is currently difficult to distinguish whether the improvement comes from new conditions, structural changes, or transfer brought about by multi-task joint training. This point is also related to contribution localization. EgoX already has conditional stitching in the width direction [1], VACE unifies multiple conditional and masking tasks [2], and Syn2Seq-Forcing involves bidirectional view generation [3]. The valuable difference in this paper lies in the combination of simultaneous multi-view, unified seven-task and geometric design. It needs to be demonstrated which single-task solutions bring benefits that cannot be obtained by this combination, and whether the performance of some tasks is sacrificed. It is recommended to prioritize increasing the single-task and joint training comparison of budget matching, and then ablate view type embedding and temporal bias respectively, and report the quality, latency and memory overhead under different number of views.
-
The statistical support for consistency evaluation and human evaluation is weak. FVD and image metrics are reported on 420 groups, but Align only covers 20 groups, including 47 first-person problems and 34 third-person problems. Appendix E (pages 24-25) also states that only 10 sets of human evaluations were used for Extension and Interpolation; this exception should be explained in the main text regarding the 20 sets of evaluation samples.
Table 1 does not provide confidence intervals for the differences. Text2Ego's Overall score is 3.68 vs. 3.66, making it difficult to confirm a stable improvement in human preference; Ego2Exo's 3.71 is also lower than A14B's 3.78. The overall trend is positive, but the reliability of all differences cannot be judged solely by the mean.
Align asks questions separately for each target perspective and then averages them, primarily reflecting semantic event matching, but not directly examining whether the actions, contact relationships, and spatial layouts of multiple generated perspectives are consistent at the same time. The lack of video filtering and time reversal comparisons is helpful, but the problem selection relies on the same evaluation model and is only called once; its stability still needs evaluation. The correlation between cross-method and cross-task means cannot replace sample-level validation.
It is recommended to expand the independent take samples, provide paired confidence intervals and task-specific correlations, and add direct cross-view synchronous evaluation.
Pixel and feature metrics should specify whether conditional frames were excluded; when generating multiple external views, it should be clearly stated whether all views are evaluated or only one view is selected.
Questions for authors
-
Are the interpolation anchor points obtained from the complete video encoding containing unknown intermediate frames? After fixing visible frames and replacing intermediate frames, do the anchor points remain unchanged?
-
What specific trajectories, viewpoints, and observation frames are obtained for each baseline? How does the same complete model perform without inputting real trajectories?
-
Under the same data, conditions, and training budget, what are the gains or losses of the joint model compared to the single-task model?
-
After expanding the sample and providing intervals, do the advantages of Align and human evaluation still hold? What are the results of direct cross-viewpoint simultaneous evaluation?
8 comments
Comment 2/6 (W2, Q2: conditioning and trajectory-free inference)
W2. The main table uses the real ego trajectory, which in Extension includes future head motion, and the paper does not explain item by item which trajectories, input views and observed frames each baseline receives. EgoU reads up to three exo views while some baselines use one; Extension uses a nine-frame prefix; B.4 says single-panel tasks disable the geometry branch, yet the ray embedding of Eq. (6) exists independently.
The conditions each method receives are listed below. At a matched input count, EgoU remains 0.65 dB above A14B on Exo2Ego.
Trajectories. EgoU, EgoX, and Exo2Ego-V receive ground-truth head trajectories. The two adapted Wan2.2 baselines also receive head trajectories through cross-attention. WorldWander, EgoExo-Gen, Framer, SEINE and Wan2.1-14B FLF2V do not receive a trajectory. In Extension, a full trajectory supplies future head motion, so the corresponding result measures generation conditioned on that motion.
View counts. The audited A14B multi-view runs use two-panel canvases, while EgoU uses up to four panels. Additional exo inputs improve EgoU's Exo2Ego PSNR by 0.60 dB from one to three input views on the 104-group matched subset. When those panels are generation targets in Ego2Exo, exo1 PSNR decreases by 0.44 dB. So the extra panels help EgoU on Exo2Ego and cost it on Ego2Exo relative to a two-panel run. At a matched input count the Exo2Ego advantage remains: in the same unseeded V2V setting, with a single exo view and on the same 104 groups, EgoU at N=2 reaches 15.02 dB against 14.37 dB for A14B, a margin of 0.65 dB, so the PSNR advantage over A14B does not depend on the additional views.
| Method, Exo2Ego (V2V) | Panels N | exo views given | PSNR |
|---|---|---|---|
| Wan2.2-A14B | 2 | 1 | 14.37 |
| EgoU | 2 | 1 | 15.02 |
| EgoU | 3 | 2 | 15.36 |
| EgoU | 4 | 3 | 15.62 |
Extension prefix. EgoU and both adapted Wan baselines receive the first nine pixel frames of each panel, corresponding to three latent frames.
Single-panel conditioning. In the evaluated N=1 path, geometry cross-attention, ground-truth trajectory conditioning, and Plücker ray embeddings are active. The reported Text2Ego result therefore measures generation from text, a seed image, and trajectory conditioning.
Q2. How does the same complete model perform without real trajectories as input?
We evaluated Text2Ego and Extension with the trajectory supplied or withheld at inference using the same complete checkpoint. The unavailable-trajectory mode uses the all-zero track and availability flag used by training-time trajectory dropout (probability 0.4).
| Task | Condition | PSNR ↑ | SSIM ↑ | LPIPS ↓ |
|---|---|---|---|---|
| Text2Ego | Trajectory given | 14.51 | 0.524 | 0.649 |
| Text2Ego | Trajectory withheld | 14.44 | 0.531 | 0.677 |
| Extension | Trajectory given | 21.36 | 0.733 | 0.243 |
| Extension | Trajectory withheld | 21.08 | 0.723 | 0.266 |
For Text2Ego, withholding the trajectory reduces PSNR by 0.07 dB and increases LPIPS by 0.028, while SSIM increases by 0.007. For Extension, withholding the trajectory reduces PSNR by 0.28 dB and SSIM by 0.010, and increases LPIPS by 0.023. The trajectory improves all three metrics for Extension and gives mixed results for Text2Ego.
Revision. We will document the observation interface separately for each baseline and task, correct Appendix B.4 and the Figure 2 label, update the Text2Ego description, including Appendix D, to state its inputs, and add the matched-count and trajectory-free tables.
Comment 3/6 (W3, Q3: isolating unified training, component ablations, prior work)
W3, Q3. The ablation does not separate the benefit of unified training itself: there is no single-task canvas model with the same backbone, data, conditions and budget, so it is unclear what the joint model gains or loses, and whether some tasks are sacrificed. EgoX, VACE and Syn2Seq-Forcing each cover part of the design. Then ablate the view-type embedding and the temporal bias.
We ran the budget-matched single-task control and both ablations. Under identical backbone, data, conditioning and 10,000 steps, joint training costs 0.19 dB on Ego2Exo and gains 0.42 dB on Exo2Ego.
Single-task control. We trained two further canvas models that differ from the "+ Canvas Training" row of Table 2 in a single argument, the task-sampling distribution: one sees only Ego2Exo, the other only Exo2Ego. Everything else is held fixed: the same Wan2.2-TI2V-5B initialization, the same EgoExoVideo-600K training split, the same conditioning patterns and panel-count distribution, four GPUs with gradient accumulation 2, and the same 10,000-step budget. Both are evaluated on all 420 groups under the protocol of Table 2.
| Model (10,000 steps, canvas, no geometry, no pose) | Ego2Exo PSNR / SSIM / LPIPS | Exo2Ego PSNR / SSIM / LPIPS |
|---|---|---|
| Single-task (new control) | 23.78 / 0.840 / 0.092 | 13.87 / 0.491 / 0.637 |
| Multi-task (Table 2, + Canvas Training) | 23.59 / 0.836 / 0.095 | 14.29 / 0.491 / 0.614 |
Sharing one model costs Ego2Exo 0.19 dB and gains Exo2Ego 0.42 dB and 0.023 LPIPS (SSIM is unchanged at 0.491); the direction that gains is the one in which no pixel of the target view is given. With geometry and pose conditioning added, the complete model (Table 2, last row) is 0.72 dB and 1.48 dB above the single-task models on the two directions.
Component ablations. We trained two arms at the same 10,000-step budget, changing one thing each: one with the view-type embedding removed, one with the temporal bias on the geometry cross-attention removed. Everything else (backbone, data, conditioning, schedule, seed) is identical. All three models are scored on the same 420 groups.
| Configuration | Ego2Exo PSNR / SSIM / LPIPS | Exo2Ego PSNR / SSIM / LPIPS |
|---|---|---|
| complete model | 24.50 / 0.850 / 0.081 | 15.35 / 0.528 / 0.567 |
| − view-type embedding | 24.28 / 0.842 / 0.087 | 14.68 / 0.511 / 0.592 |
| − temporal bias (geometry) | 24.36 / 0.844 / 0.086 | 15.00 / 0.509 / 0.602 |
Removing either component lowers every metric in both directions. Without the view-type embedding the model loses 0.22 dB PSNR, 0.008 SSIM and 0.006 LPIPS on Ego2Exo, and 0.67 dB, 0.017 SSIM and 0.025 LPIPS on Exo2Ego. Without the temporal bias on the geometry cross-attention it loses 0.14 dB, 0.006 SSIM and 0.005 LPIPS on Ego2Exo, and 0.35 dB, 0.019 SSIM and 0.035 LPIPS on Exo2Ego.
Positioning relative to prior work. We agree the width-wise conditional stitching of EgoX, the unified conditioning and masking of VACE, and bidirectional view generation in Syn2Seq-Forcing each overlap one axis of this design. Our contribution is the combination (simultaneous multi-view panels, seven families under one conditioning interface, and geometry plus trajectory conditioning that is shared across all of them), together with the measurement of what it costs individual tasks in the single-task control above.
Revision. We will add the single-task control and both ablation rows to the paper, and word the positioning against EgoX, VACE and Syn2Seq-Forcing as above.
Comment 1/6 (W1, Q1: the interpolation anchors)
Thank you for the detailed and constructive review. It identified an error in how we extracted the interpolation anchors, which is fixed in this comment, and your other suggestions led to the single-task control, the component ablations, the trajectory-free test, the cost measurements and the enlarged evaluation in the comments that follow.
W1, Q1. The interpolation anchors are taken from an encode of the full clip with a causal VAE, so the last anchor may contain information about the intermediate frames to be generated. Fix the visible frames, replace or shuffle the hidden ones and check whether the anchors change; then re-evaluate with an encoding that depends only on visible pixels. Figure 16 describes the condition as the first frame and the last four; clarify which frames each baseline uses and whether scoring covers the same interval.
Your diagnosis is correct: the anchor latents as originally extracted did carry information about the hidden frames. We reproduced your reasoning, quantified the effect, fixed it and re-evaluated. The corrected Interpolation row is 22.13 dB PSNR on all 420 groups, and EgoU remains ahead of the three baselines below on all four metrics.
The mechanism. The Wan VAE is causal, so latent index 9 has a receptive field reaching back over pixel frames 1–32, exactly the span the interpolation task withholds. Encoding the clip up to the last anchor and slicing latent 9, as Appendix B.2 described, therefore produced an anchor that was a function of frames the model was supposed to predict. This is a property of the encoder's receptive field; it is not affected by the VAE being frozen.
How large it is. We ran the probe you propose on the released checkpoint: fix the visible frames (0 and 33–36), replace the hidden frames 1–32 either with a shuffled copy or with a constant gray, then re-extract the anchors and re-run interpolation (50 groups, all else identical). Had the anchors been independent of the hidden frames, the three arms would coincide. Before the fix they differ by 0.70 dB under shuffling and 0.83 dB under gray.
The fix. The sparse-anchor path now encodes only the pixel frames that the task declares visible. For S = {0, f−1} it encodes 5 frames (0, 33, 34, 35, 36) instead of 37. A unit test asserts, over 13 anchor patterns, that every frame handed to the encoder belongs to the declared visible set.
Verification. After the fix, the same three-arm probe on the same 50 groups returns the same value in all three arms: the anchors no longer depend on the hidden frames.
After the fix the three arms are identical to four decimal places on every one of the 50 groups and at every canvas width, on SSIM and LPIPS as well as PSNR. On these 50 groups the fix lowers PSNR from 22.808 to 22.125, a drop of 0.68 dB.
What changes in the paper. The correction affects only patterns whose observed latent indices are non-contiguous, which among the seven families is interpolation alone. For
all,firstandprefix_k, a latent prefix of the full encode is by causality exactly the encode of the corresponding pixel prefix, so those rows are bit-identical before and after the fix. Table 1 therefore still reports a single checkpoint, and only the Interpolation row moves. The corrected row, measured on all 420 groups from the released checkpoint, is:The baselines in this table are re-measured through the same pipeline of encoding 5 frames (0, 33, 34, 35, 36), so all rows are measured under the same protocol.
Figure 16. Under the causal grouping, latent frame 9 corresponds to pixel frames 33–36, so "the first frame and the last four" is the pixel-level description of S = {0, f−1}: five visible pixel frames, entering the model as one anchor latent at each end.
Scoring interval. Scoring covers only the generated interval: conditioned frames are excluded from every pixel and feature metric, for our model and for every baseline, and all methods receive the same visible frames.
Revision. We will correct the anchor extraction in Appendix B.2, replace the Interpolation row of Table 1 with the corrected row, and make the latent-to-pixel correspondence explicit in the caption of Figure 16. At the meanwhile, all interpolation results reported in the rebuttal, including human study and re-calculation of Align score, use videos regenerated with the corrected visible-only anchor extraction.