- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper addresses the task of Exo2Ego video generation in a streaming setup.
The task is not absolutely novel, similar works in novel-view synthesis [1] already propose solutions for a streaming setup. However I do acknowledge that the exo2ego scenario is far more complex. While distilling a bidirectional model into a chunk-wise autoregressive generator has become a standard pipeline in the literature, the authors design and propose several solutions to target various problems when it comes to exo2ego rendering.
The paper starts defining a clear separation of the visual ego view into three regions (H, S, U) which will be separately addressed by the following contributions:
- dual branch ego and pose prior:
- ego prior: provides an “ego-aligned” view built from a 3D world cache of the static environment + the dynamics of the ego agent at the target frame t
- pose prior: estimated skeletal pose (including hands) projected both on the exocentric and egocentric views.
- region-aware loss schedule: which gradually shifts learning priority from S->U->H.
- region-aware corruption: the authors cleverly notice how the exposure bias in the teacher forcing is particularly high for region H and address it via noise injection on previous frames over H.
- first-chunk curriculum similarly tackles error propagation from U by spending initial training on first-chunk conditioned predictions. Baselines comparison cover previous works on Exo2Ego task (V2V) and camera controlled models. Ablation studies cover each component discussed in the contributions.
My overall judgment of this work is positive and leaning to acceptance. While each proposed component is not per se really novel, the authors designed clever mechanisms to tackle several different issues in this task. Finally, this work is a good step towards high quality real time exo to ego viewpoint conversion which has interesting applications.
References: [1] Quesado et al. “LiveStre4m: Feed-Forward Live Streaming of Novel Views from Unposed Multi-View Video” (CVPR 26 workshop)
Critical strengths
- This is the first work proposing a solution to streaming exo2ego generation. Distilling chunk-wise causal autoregressive models is a standard pipeline nowadays, however the proposed additional mechanisms (region-aware corruption and first-chunk curriculum) are a clever way to address challenges specific to exo2ego.
- The paper is overall well written and easy to follow. Experiments and ablations cover pretty much all the useful comparison and design choices. Particularly appreciated is the level of details provided in the appendix, facilitating reproducibility.
Critical weaknesses
- Figure 10 shows an online VLM captioning system and section D.2 describes how these captions are generated and used at training time. The impact of the additional information provided by the VLM at inference time is never discussed and would need an ablation.
- The main paper overlooks the fact that the ego camera trajectory is given and briefly discusses it in the appendix. This is a crucial detail because it breaks the whole point of a streaming exo2ego end-to-end system. This was acceptable for previous "offline" works but this task is different.
Questions for authors
- Can the authors show how much the online generated vlm captions impact the quality of the generations? This is useful to understand how much the model relies on the exocentric view and priors vs the text description.
- Can the authors provide real end2end pipeline results by relying on SOTA head/pose estimators? Metrics are expected to be lower, which is fine, but the key aspect is latency. The current end2end pipeline uses privileged information that is not available at inference time in real scenarios.
3 comments
Our response to Weakness 2 and Question 2
(Weakness-2 & Question-2) Following the suggestion, we have additionally implemented and evaluated a fully end-to-end pipeline using online exocentric head tracking without any ground-truth trajectory inputs, maintaining the exact same 24.3 FPS real-time throughput. Specifically, we replace the ground-truth trajectories with a causal tracker built from open-source estimators: Multi-HMR (Baradel et al., 2024) estimates the 3D head position on every second frame, MoGe-2 (Wang et al., 2025) aligns its metric depth, SemiUHPE (Zhou et al., 2025) estimates the head orientation, and a causal One Euro filter (Casiez et al., 2012) smooths the pose trajectory; a VLM identifies the wearer once at the start.
Latency. Running the causal head tracker on one additional A100 GPU in parallel with the generator takes 0.91 s per 1.375-s chunk (33 frames at 832×480 resolution), which is well within the chunk duration and keeps the end-to-end pipelined throughput at 24.3 FPS. The per-chunk latency breakdown across each stage of the tracker is detailed below:
| Stage | Hardware | Time per chunk |
|---|---|---|
| Multi-HMR on 17 of 33 frames (head position) | A100 | 0.68 s |
| SemiUHPE on the head crops (head orientation) | A100 | 0.03 s |
| MoGe-2 on 3 frames (metric depth) | A100 | 0.19 s |
| Wearer tracking, One Euro filter | CPU | < 0.01 s |
| Head tracking in total | – | 0.91 s |
Quality. The estimated poses deviate from the dataset trajectories by 52.8 cm and 22.3° on New Action, by 54.5 cm and 22.7° on Unseen (mostly along the viewing ray, as the wearers stand about 3 m from the camera), and by 6.3 cm and 10.9° on H2O (about 1 m). Under the protocol of Ours (long), our student achieves the following results (with the ground-truth trajectory results included as a reference):
| SPLIT | EGO TRAJECTORY | S-PSNR ↑ | S-MSSIM ↑ | S-LPIPS ↓ | U-CLIP-I ↑ | [email protected] ↑ | MC ↑ | PSNR ↑ | SSIM ↑ | LOC ERR ↓ | IOU ↑ | FVD ↓ | TF ↑ | MS ↑ | FPS ↑ |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| New Action | Dataset ground truth | 18.16 | 0.588 | 0.365 | 0.893 | 47.59% | 53.48% | 17.47 | 0.588 | 0.118 | 0.447 | 135.62 | 0.981 | 0.991 | 24.3 |
| New Action | Estimated from exo video | 16.11 | 0.469 | 0.491 | 0.865 | 30.98% | 36.47% | 15.26 | 0.450 | 0.164 | 0.308 | 343.55 | 0.973 | 0.982 | 24.3 |
| Unseen | Dataset ground truth | 17.88 | 0.569 | 0.394 | 0.890 | 37.22% | 41.70% | 16.84 | 0.537 | 0.119 | 0.427 | 144.33 | 0.988 | 0.994 | 24.3 |
| Unseen | Estimated from exo video | 15.28 | 0.467 | 0.517 | 0.848 | 23.76% | 31.32% | 14.70 | 0.443 | 0.183 | 0.290 | 368.42 | 0.972 | 0.979 | 24.3 |
| H2O | Dataset ground truth | 23.21 | 0.713 | 0.118 | 0.929 | 81.80% | 89.71% | 21.05 | 0.680 | 0.056 | 0.814 | 94.17 | 0.993 | 0.994 | 24.3 |
| H2O | Estimated from exo video | 20.28 | 0.668 | 0.179 | 0.871 | 63.50% | 71.26% | 18.56 | 0.615 | 0.115 | 0.744 | 174.01 | 0.975 | 0.984 | 24.3 |
Although the estimated poses are less accurate than the ground truth trajectories, our student remains competitive with the baseline methods in Table 1, which all use the ground truth trajectories. In practical deployments equipped with dedicated hardware trackers (e.g., on-device SLAM or inertial sensors), pose accuracy and thus generation quality can be further improved. We have added this experiment to the main paper as Sec. 4.4 (L502–528).
Dear Reviewer 9ux3,
Thank you again for your positive and thoughtful review. We truly appreciate your recognition of our region-aware corruption and first-chunk curriculum, and your suggestion to evaluate a real end-to-end pipeline has made our system more complete. We understand that this is a very busy time, and we are grateful for the effort you have already put into our paper.
As the discussion period is coming to an end, we would like to briefly summarize our responses. We have also uploaded a revised version of the paper with the new experiments.
- End-to-end pipeline: We replaced the ground-truth trajectories with a causal head tracker built from open-source estimators. It takes 0.91 s per 1.375-s chunk on one additional GPU, so the full pipeline keeps its 24.3 FPS throughput, and our student remains competitive with baselines that use the ground-truth trajectories. This experiment is now in the main paper (Sec. 4.4 and Table 1).
- VLM captions: The VLM sees only the current exocentric chunk. Our ablation over empty, concise (ours) and detailed captions shows that the appearance in and relies primarily on the exocentric video and priors, while action semantics provide supplementary guidance for hand-object interactions in . Detailed scene captions bring no further benefit (Appendix B.5, Table 6).
We would greatly appreciate any further comments or suggestions you may have, which would help us further improve the paper. If our responses have addressed your concerns, we would be grateful if you could take them into account in your final assessment.
Best regards, The Authors
Our response to Weakness 1 and Question 1
We sincerely thank the reviewer for the positive assessment and the thoughtful questions. Line numbers below refer to the revised PDF.
(Weakness-1 & Question-1) The online VLM observes only the current exocentric chunk therefore it introduces no external information beyond the exocentric video. Moreover, as described in Appendix D.2, we apply PromptMix regularization during training to prevent the model from over-relying on text conditioning. To quantify the actual impact of inference-time captions versus the exocentric video and priors, we evaluate the student on Unseen across three caption detail levels generated by the same online VLM: (1) an empty string; (2) a description of the hand action and manipulated object (ours); and (3) the manipulation description from (2) augmented with a detailed scene summary. In all variants, the VLM still identifies dynamic objects for SAM 3.1 segmentation, isolating only the effect of the generator's text prompt:
The results confirm that visual appearance in S and U relies primarily on the exocentric video and geometric priors rather than text, whereas action semantics from the VLM provide supplementary guidance for hand-object interactions in H. Adding detailed scene descriptions in (3) brings no further benefit over our concise prompt, showing that the model does not need scene captions from the VLM. We have added this ablation to Appendix B.5 (L1018–1045, Table 6).