acceptodds
Under review as a conference paper at ICLR 2027

StreamEgo: Region-Aware Autoregressive Exocentric-to-Egocentric Video Generation

14 views

Abstract

Exocentric-to-egocentric (Exo2Ego) video generation transforms third-person recordings into first-person viewpoints, unlocking immersive AR/VR experiences and scalable demonstration synthesis for embodied robotics. Extending it to a streaming setting is expected to further enable live applications. However, existing methods suffer from degraded manipulation regions, overfitting and confinement to fixed-length clips with prohibitive latency. To address these limitations, we present StreamEgo, a real-time framework for unbounded, generalizable Exo2Ego video generation. To resolve intricate manipulation dynamics while supporting the streaming pipeline, a streaming dual-branch prior provides pixel-aligned guidance for translating appearance and skeletal kinematics. We also introduce a region-aware loss curriculum to improve generalization and stabilize multi-objective diffusion training. Furthermore, we distill the model into a chunk-wise autoregressive model, augmented with a region-aware corruption mechanism and a first-chunk curriculum, enabling high-quality, generalizable real-time streaming Exo2Ego generation at 24.3 FPS on a single H100 GPU.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Exocentric-to-Egocentric Video Generation, Video Diffusion Models, Autoregressive Generation, Human–Object Interaction

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_9ux3OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the task of Exo2Ego video generation in a streaming setup.

The task is not absolutely novel, similar works in novel-view synthesis [1] already propose solutions for a streaming setup. However I do acknowledge that the exo2ego scenario is far more complex. While distilling a bidirectional model into a chunk-wise autoregressive generator has become a standard pipeline in the literature, the authors design and propose several solutions to target various problems when it comes to exo2ego rendering.

The paper starts defining a clear separation of the visual ego view into three regions (H, S, U) which will be separately addressed by the following contributions:

  • dual branch ego and pose prior:
    • ego prior: provides an “ego-aligned” view built from a 3D world cache of the static environment + the dynamics of the ego agent at the target frame t
    • pose prior: estimated skeletal pose (including hands) projected both on the exocentric and egocentric views.
  • region-aware loss schedule: which gradually shifts learning priority from S->U->H.
  • region-aware corruption: the authors cleverly notice how the exposure bias in the teacher forcing is particularly high for region H and address it via noise injection on previous frames over H.
  • first-chunk curriculum similarly tackles error propagation from U by spending initial training on first-chunk conditioned predictions. Baselines comparison cover previous works on Exo2Ego task (V2V) and camera controlled models. Ablation studies cover each component discussed in the contributions.

My overall judgment of this work is positive and leaning to acceptance. While each proposed component is not per se really novel, the authors designed clever mechanisms to tackle several different issues in this task. Finally, this work is a good step towards high quality real time exo to ego viewpoint conversion which has interesting applications.

References: [1] Quesado et al. “LiveStre4m: Feed-Forward Live Streaming of Novel Views from Unposed Multi-View Video” (CVPR 26 workshop)

Critical strengths
  1. This is the first work proposing a solution to streaming exo2ego generation. Distilling chunk-wise causal autoregressive models is a standard pipeline nowadays, however the proposed additional mechanisms (region-aware corruption and first-chunk curriculum) are a clever way to address challenges specific to exo2ego.
  2. The paper is overall well written and easy to follow. Experiments and ablations cover pretty much all the useful comparison and design choices. Particularly appreciated is the level of details provided in the appendix, facilitating reproducibility.
Critical weaknesses
  1. Figure 10 shows an online VLM captioning system and section D.2 describes how these captions are generated and used at training time. The impact of the additional information provided by the VLM at inference time is never discussed and would need an ablation.
  2. The main paper overlooks the fact that the ego camera trajectory is given and briefly discusses it in the appendix. This is a crucial detail because it breaks the whole point of a streaming exo2ego end-to-end system. This was acceptable for previous "offline" works but this task is different.
Questions for authors
  1. Can the authors show how much the online generated vlm captions impact the quality of the generations? This is useful to understand how much the model relies on the exocentric view and priors vs the text description.
  2. Can the authors provide real end2end pipeline results by relying on SOTA head/pose estimators? Metrics are expected to be lower, which is fine, but the key aspect is latency. The current end2end pipeline uses privileged information that is not available at inference time in real scenarios.
3 comments
AuthorsOpenReview ↗

Our response to Weakness 1 and Question 1

We sincerely thank the reviewer for the positive assessment and the thoughtful questions. Line numbers below refer to the revised PDF.

(Weakness-1 & Question-1) The online VLM observes only the current exocentric chunk therefore it introduces no external information beyond the exocentric video. Moreover, as described in Appendix D.2, we apply PromptMix regularization during training to prevent the model from over-relying on text conditioning. To quantify the actual impact of inference-time captions versus the exocentric video and priors, we evaluate the student on Unseen across three caption detail levels generated by the same online VLM: (1) an empty string; (2) a description of the hand action and manipulated object (ours); and (3) the manipulation description from (2) augmented with a detailed scene summary. In all variants, the VLM still identifies dynamic objects for SAM 3.1 segmentation, isolating only the effect of the generator's text prompt:

Text prompt at inferenceS-PSNR ↑S-MSSIM ↑S-LPIPS ↓U-CLIP-I ↑[email protected] ↑MC ↑Loc Err ↓IoU ↑FVD ↓TF ↑MS ↑
(1) Empty17.860.5750.4150.86626.60%30.97%0.1310.413173.530.9880.993
(2) Concise (ours)17.880.5690.3940.89037.22%41.70%0.1190.427144.330.9880.994
(3) Detailed17.910.5720.3900.89437.51%39.01%0.1270.423147.650.9880.993

The results confirm that visual appearance in S\mathcal{S} and U\mathcal{U} relies primarily on the exocentric video and geometric priors rather than text, whereas action semantics from the VLM provide supplementary guidance for hand-object interactions in H\mathcal{H}. Adding detailed scene descriptions in (3) brings no further benefit over our concise prompt, showing that the model does not need scene captions from the VLM. We have added this ablation to Appendix B.5 (L1018–1045, Table 6).

AuthorsOpenReview ↗

Our response to Weakness 2 and Question 2

(Weakness-2 & Question-2) Following the suggestion, we have additionally implemented and evaluated a fully end-to-end pipeline using online exocentric head tracking without any ground-truth trajectory inputs, maintaining the exact same 24.3 FPS real-time throughput. Specifically, we replace the ground-truth trajectories with a causal tracker built from open-source estimators: Multi-HMR (Baradel et al., 2024) estimates the 3D head position on every second frame, MoGe-2 (Wang et al., 2025) aligns its metric depth, SemiUHPE (Zhou et al., 2025) estimates the head orientation, and a causal One Euro filter (Casiez et al., 2012) smooths the pose trajectory; a VLM identifies the wearer once at the start.

Latency. Running the causal head tracker on one additional A100 GPU in parallel with the generator takes 0.91 s per 1.375-s chunk (33 frames at 832×480 resolution), which is well within the chunk duration and keeps the end-to-end pipelined throughput at 24.3 FPS. The per-chunk latency breakdown across each stage of the tracker is detailed below:

StageHardwareTime per chunk
Multi-HMR on 17 of 33 frames (head position)A1000.68 s
SemiUHPE on the head crops (head orientation)A1000.03 s
MoGe-2 on 3 frames (metric depth)A1000.19 s
Wearer tracking, One Euro filterCPU< 0.01 s
Head tracking in total–0.91 s

Quality. The estimated poses deviate from the dataset trajectories by 52.8 cm and 22.3° on New Action, by 54.5 cm and 22.7° on Unseen (mostly along the viewing ray, as the wearers stand about 3 m from the camera), and by 6.3 cm and 10.9° on H2O (about 1 m). Under the protocol of Ours (long), our student achieves the following results (with the ground-truth trajectory results included as a reference):

SPLITEGO TRAJECTORYS-PSNR ↑S-MSSIM ↑S-LPIPS ↓U-CLIP-I ↑[email protected] ↑MC ↑PSNR ↑SSIM ↑LOC ERR ↓IOU ↑FVD ↓TF ↑MS ↑FPS ↑
New ActionDataset ground truth18.160.5880.3650.89347.59%53.48%17.470.5880.1180.447135.620.9810.99124.3
New ActionEstimated from exo video16.110.4690.4910.86530.98%36.47%15.260.4500.1640.308343.550.9730.98224.3
UnseenDataset ground truth17.880.5690.3940.89037.22%41.70%16.840.5370.1190.427144.330.9880.99424.3
UnseenEstimated from exo video15.280.4670.5170.84823.76%31.32%14.700.4430.1830.290368.420.9720.97924.3
H2ODataset ground truth23.210.7130.1180.92981.80%89.71%21.050.6800.0560.81494.170.9930.99424.3
H2OEstimated from exo video20.280.6680.1790.87163.50%71.26%18.560.6150.1150.744174.010.9750.98424.3

Although the estimated poses are less accurate than the ground truth trajectories, our student remains competitive with the baseline methods in Table 1, which all use the ground truth trajectories. In practical deployments equipped with dedicated hardware trackers (e.g., on-device SLAM or inertial sensors), pose accuracy and thus generation quality can be further improved. We have added this experiment to the main paper as Sec. 4.4 (L502–528).

Dear Reviewer 9ux3,

Thank you again for your positive and thoughtful review. We truly appreciate your recognition of our region-aware corruption and first-chunk curriculum, and your suggestion to evaluate a real end-to-end pipeline has made our system more complete. We understand that this is a very busy time, and we are grateful for the effort you have already put into our paper.

As the discussion period is coming to an end, we would like to briefly summarize our responses. We have also uploaded a revised version of the paper with the new experiments.

  • End-to-end pipeline: We replaced the ground-truth trajectories with a causal head tracker built from open-source estimators. It takes 0.91 s per 1.375-s chunk on one additional GPU, so the full pipeline keeps its 24.3 FPS throughput, and our student remains competitive with baselines that use the ground-truth trajectories. This experiment is now in the main paper (Sec. 4.4 and Table 1).
  • VLM captions: The VLM sees only the current exocentric chunk. Our ablation over empty, concise (ours) and detailed captions shows that the appearance in S\mathcal{S} and U\mathcal{U} relies primarily on the exocentric video and priors, while action semantics provide supplementary guidance for hand-object interactions in H\mathcal{H}. Detailed scene captions bring no further benefit (Appendix B.5, Table 6).

We would greatly appreciate any further comments or suggestions you may have, which would help us further improve the paper. If our responses have addressed your concerns, we would be grateful if you could take them into account in your final assessment.

Best regards, The Authors

Reviewer_jYXgOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This work proposes a novel framework for Exo2Ego generation that enables real-time, streamable generation while improving generalizability and interaction fidelity. To enable real-time, streamable generation, the authors adopt a teacher-forcing-based distillation method to obtain a causal, chunk-wise autoregressive student model. To mitigate exposure bias and motion drift in the distilled student model, they propose region-aware corruption and a first-chunk curriculum to improve robustness. Furthermore, to improve generalizability, the authors propose using skeletal constraints and a region-aware training curriculum.

Critical strengths

The work proposes novel methods (region-aware curriculum learning, region-aware corruption, and a first-chunk curriculum) that are well suited to Exo2Ego generation. These methods appear plausible and are expected to improve upon previous methods.

Critical weaknesses
  • It is difficult to evaluate the effectiveness of the proposed methods because the qualitative ablation results are insufficient. While comparisons with previous methods are presented across approximately 10 different scenarios, along with videos in the supplementary materials, the ablation studies are presented for only one scenario, with no accompanying video.
  • While the authors claim that region-aware curriculum learning mitigates benchmark memorization, the evidence supporting this claim is limited. To validate this claim, Table 2 should also report results on both the NEW ACTION and UNSEEN benchmarks. If the region-aware curriculum mitigates benchmark memorization, the metrics for UNSEEN benchmark are expected to be similar to those on the NEW ACTION benchmark, while the metrics for unseen regions are expected to deteriorate.
Questions for authors
  • In Table 1, why do the proposed approaches achieve better scores on H2O than on UNSEEN and NEW ACTION? Since the models are trained on EgoExo4D, they are expected to perform better on UNSEEN and NEW ACTION, which are within the EgoExo4D distribution. While the baselines achieve worse or similar scores on H2O, only the proposed methods achieve better scores on H2O than on UNSEEN and NEW ACTION.
  • How are the extrinsic poses of the exocentric and egocentric cameras obtained in real-world scenarios? If these poses are difficult to obtain in practice, this should be discussed in the limitations section.
  • Since EgoX and the proposed method use different priors and backbones (Wan 2.1 14B and SAM2 vs. Wan2.2-TI2V-5B and SAM3.1), the authors are encouraged to report an ablation study on the priors used in their method.
  • Addressing the concerns raised in the Weaknesses and Questions sections would help me re-evaluate my assessment of this work.
5 comments
AuthorsOpenReview ↗

Our response to Weakness 1-2 and Question 1

We thank the reviewer for the careful review and the constructive suggestions. Line numbers below refer to the revised PDF.

(Weakness-1) In addition to Figure 6, Figure 7 already presents qualitative ablations of the student across two additional scenarios. Following the reviewer's suggestion, we have added four more examples of the teacher ablations in Figure 6 to the Appendix (Figure 9, Appendix B.1, L943–944). Our qualitative ablations therefore cover seven different examples in total. The ablation videos were omitted from the supplementary material solely due to the file size limit, and we will release our code along with all qualitative videos in our final version.

(Weakness-2) Table 2 is already evaluated on Unseen (we have revised the paper to make this clear in L448–449). Following the suggestion, we have added the region-aware curriculum ablation on New Action to Table 2 (w/o HSU):

VariantSplitS-PSNR ↑S-MSSIM ↑S-LPIPS ↓U-CLIP-I ↑[email protected] ↑MC ↑Loc Err ↓IoU ↑FVD ↓TF ↑MS ↑
Full teacherNew Action18.350.6030.3720.90749.86%56.26%0.0760.478111.970.9830.994
w/o HSUNew Action17.640.5310.4650.90235.49%42.70%0.1500.291144.970.9760.992
Full teacherUnseen18.180.5760.4050.90140.24%43.76%0.1060.456138.910.9800.995
w/o HSUUnseen16.880.4810.5140.86429.22%31.37%0.2220.139301.760.9770.989

We point out that the pattern expected by the reviewer is already reflected in Table 1: our method achieves comparable S\mathcal{S}-PSNR, S\mathcal{S}-MSSIM, and S\mathcal{S}-LPIPS across New Action and Unseen, whereas full-image PSNR and SSIM drop on Unseen. This indicates lower pixel-level agreement in the unobservable region U\mathcal{U}, while U\mathcal{U}-CLIP-I remains consistent across both splits, showing that our model performs plausible, semantic-preserving inpainting in U\mathcal{U} rather than memorizing training scenes.

The expanded ablation in Table 2 (w/o HSU) further confirms that this generalization stems from our region-aware curriculum: without it, the performance exhibits a pronounced gap between New Action and Unseen, reflecting overfitting to seen environments. This behavior is also supported by the training loss curves in Figure 6, where removing the curriculum causes U\mathcal{U} loss to overfit early while slowing convergence in S\mathcal{S} and H\mathcal{H}. We have added this discussion to Sec. 4.3 (L480–487).

(Question-1) StreamEgo scores higher on H2O because H2O presents a geometrically easier view-transfer setup than Ego-Exo4D, and most baselines actually exhibit the same trend.

First, across the revised Table 1, baselines also achieve higher scores on H2O than on Unseen in 45 out of 65 (~70%) metric entries, and Vista4D (which similarly conditions on rendered geometry) gains even more than ours.

Second, although H2O differs from Ego-Exo4D in hardware, subjects, and scenes (Appendix A.3), its capture geometry makes view transfer inherently easier (Figures 4, 5, and 8 and the supplementary video): (1) the exocentric cameras are much closer to the subject (~1 m vs. ~3 m); (2) the exocentric view is mostly unoccluded and closer in direction to the egocentric view, leaving a smaller unseen region U\mathcal{U} and yielding more accurate priors; (3) the objects and tablecloths have simpler shapes and regular patterns; and (4) the actions are slower.

We argue that methods that exploit the exocentric evidence therefore gain on H2O, whereas a lower score would rather indicate reliance on the training distribution.

AuthorsOpenReview ↗

Our response to Question 2

(Question-2) In real-world scenarios, the world frame can be anchored directly at the stationary exocentric camera (eliminating the need for exocentric extrinsics), while the egocentric camera trajectory can be estimated online from the exocentric video via 3D head pose tracking (Appendix E, L1443–1450). Consistent with prior Exo2Ego works, we take the egocentric trajectory as a given input in our main formulation (L132–134) and have already discussed this practical constraint in our Limitations section (Appendix E, L1431–1450).

To test our method without this assumption, we replace them with head tracking from the exocentric camera and run the full pipeline end to end. The tracker is causal and built from open-source estimators: Multi-HMR (Baradel et al., 2024) estimates the 3D head position on every second frame, MoGe-2 (Wang et al., 2025) aligns its metric depth, SemiUHPE (Zhou et al., 2025) estimates the head orientation, and a causal One Euro filter (Casiez et al., 2012) smooths the pose trajectory; a VLM identifies the wearer once at the start.

Latency. Running the causal head tracker on one additional A100 GPU in parallel with the generator takes 0.91 s per 1.375-s chunk (33 frames at 832×480 resolution), which is well within the chunk duration and keeps the end-to-end pipelined throughput at 24.3 FPS. The per-chunk latency breakdown across each stage of the tracker is detailed below:

StageHardwareTime per chunk
Multi-HMR on 17 of 33 frames (head position)A1000.68 s
SemiUHPE on the head crops (head orientation)A1000.03 s
MoGe-2 on 3 frames (metric depth)A1000.19 s
Wearer tracking, One Euro filterCPU< 0.01 s
Head tracking in total–0.91 s

Quality. The estimated poses deviate from the dataset trajectories by 52.8 cm and 22.3° on New Action, by 54.5 cm and 22.7° on Unseen (mostly along the viewing ray, as the wearers stand about 3 m from the camera), and by 6.3 cm and 10.9° on H2O (about 1 m). Under the protocol of Ours (long), our student achieves the following results (with the ground-truth trajectory results included as a reference):

SPLITEGO TRAJECTORYS-PSNR ↑S-MSSIM ↑S-LPIPS ↓U-CLIP-I ↑[email protected] ↑MC ↑PSNR ↑SSIM ↑LOC ERR ↓IOU ↑FVD ↓TF ↑MS ↑FPS ↑
New ActionDataset ground truth18.160.5880.3650.89347.59%53.48%17.470.5880.1180.447135.620.9810.99124.3
New ActionEstimated from exo video16.110.4690.4910.86530.98%36.47%15.260.4500.1640.308343.550.9730.98224.3
UnseenDataset ground truth17.880.5690.3940.89037.22%41.70%16.840.5370.1190.427144.330.9880.99424.3
UnseenEstimated from exo video15.280.4670.5170.84823.76%31.32%14.700.4430.1830.290368.420.9720.97924.3
H2ODataset ground truth23.210.7130.1180.92981.80%89.71%21.050.6800.0560.81494.170.9930.99424.3
H2OEstimated from exo video20.280.6680.1790.87163.50%71.26%18.560.6150.1150.744174.010.9750.98424.3

Although the estimated poses are less accurate than the ground truth trajectories, our student remains competitive with the baseline methods in Table 1, which all use the ground truth trajectories. Our results can further improve as head tracking becomes more accurate. We have added this experiment to the main paper as Sec. 4.4 (L502–528).

AuthorsOpenReview ↗

Our response to Question 3

(Question-3) We note that the E prior row in Table 2 (L464–465, L473–475) already ablates the effect of the prior and segmentation method, as it replaces our dynamic ego prior (using SAM 3.1) with the offline ego prior of EgoX (using SAM-ViT-B) while keeping the same human pose prior.

Following the reviewer's suggestion, we also conducted an ablation on the backbone by porting EgoX from Wan2.1-14B to our Wan2.2-TI2V-5B backbone, keeping its original ego prior, conditioning, and GGA unchanged. We fine-tuned it on our training set under the same resolution, clip length, and frame rate as our model, and evaluated it on the same protocol as Table 1:

SplitMethodBackboneS-PSNR ↑S-MSSIM ↑S-LPIPS ↓U-CLIP-I ↑[email protected] ↑MC ↑PSNR ↑SSIM ↑Loc Err ↓IoU ↑FVD ↓TF ↑MS ↑FPS ↑
New ActionEgoX (Table 1)Wan2.1-I2V-14B (orig.)15.750.4890.5800.88825.91%34.84%16.060.5610.1300.357184.240.9770.9900.08
New ActionEgoXWan2.2-TI2V-5B16.010.5050.5750.87327.06%32.42%16.310.5730.1350.339195.260.9740.9900.46
New ActionOurs (teacher)Wan2.2-TI2V-5B18.350.6030.3720.90749.86%56.26%17.510.6030.0760.478111.970.9830.9940.97
UnseenEgoX (Table 1)Wan2.1-I2V-14B (orig.)13.360.4060.5770.87911.07%15.41%14.410.4560.3110.092438.790.9810.9920.08
UnseenEgoXWan2.2-TI2V-5B13.590.4120.5540.88812.31%15.23%14.860.4480.2970.127468.690.9880.9900.46
UnseenOurs (teacher)Wan2.2-TI2V-5B18.180.5760.4050.90140.24%43.76%17.190.5440.1060.456138.910.9800.9950.97
H2OEgoX (Table 1)Wan2.1-I2V-14B (orig.)14.230.4650.3770.71035.10%40.28%15.000.4680.2050.513479.520.9110.9290.08
H2OEgoXWan2.2-TI2V-5B14.450.4760.3590.69438.58%39.37%15.520.4900.1860.505501.250.9280.9390.46
H2OOurs (teacher)Wan2.2-TI2V-5B23.590.7240.1060.93189.22%95.78%21.350.6880.0410.87689.940.9930.9940.97

The result confirms that the backbone does not account for the performance gap in Table 1. We have added this ablation to Appendix B.6 (L1049–1058, Table 7). Since the discussion period is short, this model might not have fully converged. We are actively continuing the training and updating the results throughout the discussion period.

Reviewer_wAHKOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

StreamEgo proposes the first streaming-based exo-to-ego video generation method. It achieves strong performance by training on a region-weighted loss curriculum and then distilling that model into a causal generator, closely following Helios.

Critical strengths

Streaming Exo2Ego is highly relevant, difficult to achieve and novel.

Critical weaknesses

Some description of the method is unclear:

  • Where does the wrist joint position come from in Eq 2? The skeleton is projected into egocentric view with the calibrated cameras, but for that Bt,wristsB_{t,\mathrm{wrist}}^{s} has to be in the same space as the camera - how is that done with when ViTPose is in 2D and SMPLer-X is camera-relative?

How were the baselines calculated for Table 1? This seems to be a new protocol with its own split. Were the baselines re-run on this new protocol?

The streaming video generation mostly follows Helios - the paper acknowledges this in appendix D3 (L1034) but this should be made more explicit.

The paper borrows the conditioning layout from EgoX but drops the GGA? Why? In EgoX ablations the GGA improves quality substantially!

The authors claim viewpoint consistency and “resilience to drift”. However, this is not well-studied: for example, the qualitative results (video, Figure 5) show that the viewpoint is well-handled - however, this is to be expected as the view is anchored at the gt ego pose (Eq 1), but content can still drift, i.e. objects changing shape, as can be seen in the supplementary video. I would also argue that any form of PSNR is a bad metric for measuring drift (ablation) - Maybe something like “Drifting Score” from the Helios paper could be leveraged for this?

Where do the ego trajectories come from? Can the authors comment if they are computed offline? Could the authors test with head tracking coming from the exo camera?

The authors present a generative method but do not report variance on their experiments (Table 1 and 2) - to better understand how the model behaves it is crucial to run it multiple times with the same input and report mean and std. I am willing to increase my score if this is addressed.

Questions for authors

What happens if there is a static object (i.e. a mug) which remains static for a while but then is moved. Because of the streaming setting the segmentation does not “know” the mug will move later?

What is capture-to-display latency for all stages (not just the denoiser)?

6 comments
AuthorsOpenReview ↗

Our response to Weakness 1-4

We thank the reviewer for the careful and constructive review. Line numbers below refer to the revised PDF.

(Weakness-1) In Eq. 2, Bt,wristsB^{s}_ {t,\mathrm{wrist}} is obtained in the calibrated exocentric camera frame by anchoring SMPLer-X’s root-relative 3D skeleton to the known 3D position of the egocentric camera (ctc_ t) and fitting its metric scale via 2D reprojection against ViTPose keypoints.

Specifically, we resolve the 2D and camera-relative ambiguity in three steps:

  1. Metric Anchoring: We discard SMPLer-X's unreliable camera-relative translation, retaining only its head-relative 3D offsets (B^t−B^t,head\widehat{B}_ t - \widehat{B}_ {t,\mathrm{head}}), and anchor the head to the calibrated egocentric camera center ctc_ t of Tg,tT_ {g,t} (already given as input in Eq. 1).
  2. Scale & Offset Fitting: We parameterize the metric 3D body joints as Bt=γ(B^t−B^t,head)+ct+ΔB_ t = \gamma(\widehat{B}_ t - \widehat{B}_ {t,\mathrm{head}}) + c_ t + \Delta, solving for the global metric scale γ\gamma and small head-to-camera offset Δ\Delta by minimizing the 2D reprojection error against ViTPose detections under calibrated exocentric intrinsics.
  3. Arm & Wrist Refinement: The arm joints are refined identically under bone-length constraints, and a high-confidence WiLoR detection further pulls the wrist toward the detected 2D wrist location while preserving its fitted metric depth.

Because BtB_ t now resides directly in the calibrated metric camera space, Bt,wristsB^{s}_ {t,\mathrm{wrist}} (and the resulting articulated hand joints Jt,jsJ^{s}_ {t,j}) can be projected cleanly into both views without requiring any egocentric pixels or extra inputs beyond Eq. 1. We have added this clarification to Sec. 3.1 (L195–198) and detailed the full formulation in Appendix D.1 (L1161–1173).

(Weakness-2) TrajCrafter, Wan-VACE, Vista4D, and EgoX were fine-tuned on our training set and re-run on our test splits using the same metrics as our models.

For Exo2Ego-V, whose trained weights for egocentric generation were not released, it was trained and evaluated under the protocol of EgoX to align with prior works. We have now also trained Exo2Ego-V with its released code on our training set, evaluated it on our test splits, and updated its rows in Table 1 accordingly:

SplitMethodS-PSNR ↑S-MSSIM ↑S-LPIPS ↓U-CLIP-I ↑[email protected] ↑MC ↑PSNR ↑SSIM ↑Loc Err ↓IoU ↑FVD ↓TF ↑MS ↑
New ActionExo2Ego-V13.840.4060.5580.80415.23%17.47%14.370.3930.3150.098605.900.9630.965
UnseenExo2Ego-V13.060.4250.5810.69111.57%13.26%12.940.3850.4610.0031237.550.9730.978
H2OExo2Ego-V12.250.4020.4930.67512.39%15.14%12.340.4110.4450.025882.920.9650.972

As shown, Exo2Ego-V exhibits consistent performance across both protocols, and our comparative advantages and conclusions remain unchanged.

(Weakness-3) Our main text already credits Helios (Yuan et al., 2026) in three places: L116–117, L258–259 and L282–283. Following the suggestion, we have added another explicit reference in the opening paragraph of Sec. 3.3 (L252–254).

(Weakness-4) We agree that GGA is an effective design. We dropped it only to run in real time: a chunk covers 1.375 s of video and our denoiser already takes 1.36 s, while GGA raises the denoising time of EgoX from 6.5 to 10.5 minutes.

AuthorsOpenReview ↗

Our response to Weakness 5

(Weakness-5) Following the suggestion, we evaluate the student ablations with the Drifting Score of Helios, using its official code, on the same 30-second rollouts on Unseen as Table 2. Each of its four metrics (Aesthetic, Smoothness, Semantic and Naturalness) is computed on the first and the last 15% of the frames, and their absolute difference is mapped to an integer rating from 1 to 10 with the thresholds of Helios (higher means less drift). Since an egocentric video also changes by itself within 30 seconds, we score the ground-truth videos as a reference.

VariantDrifting Aesthetic ↑Drifting Smoothness ↑Drifting Semantic ↑Drifting Naturalness ↑
Ground truth710710
Full student610710
w/o H corruption6979
w/o first-chunk4877

The full student stays close to the ground truth while removing region-aware corruption or the first-chunk curriculum increases the drift. We have added this evaluation to Appendix B.4 (L1006–1015, Table 5).

AuthorsOpenReview ↗

Our response to Weakness 6

(Weakness-6) The ego trajectories come from the datasets and are computed offline from the camera worn by the subject: Ego-Exo4D bundle-adjusts the whole recording with Aria Machine Perception Services, and H2O solves each pose from IR markers by PnP with Kalman smoothing. Like prior Exo2Ego works, we take them as given inputs (L132–134).

Following the suggestion, we replace them with head tracking from the exocentric camera and run the full pipeline end to end. The tracker is causal and built from open-source estimators: Multi-HMR (Baradel et al., 2024) estimates the 3D head position on every second frame, MoGe-2 (Wang et al., 2025) aligns its metric depth, SemiUHPE (Zhou et al., 2025) estimates the head orientation, and a causal One Euro filter (Casiez et al., 2012) smooths the pose trajectory; a VLM identifies the wearer once at the start.

Latency. Running the causal head tracker on one additional A100 GPU in parallel with the generator takes 0.91 s per 1.375-s chunk (33 frames at 832×480 resolution), which is well within the chunk duration and keeps the end-to-end pipelined throughput at 24.3 FPS. The per-chunk latency breakdown across each stage of the tracker is detailed below:

StageHardwareTime per chunk
Multi-HMR on 17 of 33 frames (head position)A1000.68 s
SemiUHPE on the head crops (head orientation)A1000.03 s
MoGe-2 on 3 frames (metric depth)A1000.19 s
Wearer tracking, One Euro filterCPU< 0.01 s
Head tracking in total–0.91 s

Quality. The estimated poses deviate from the dataset trajectories by 52.8 cm and 22.3° on New Action, by 54.5 cm and 22.7° on Unseen (mostly along the viewing ray, as the wearers stand about 3 m from the camera), and by 6.3 cm and 10.9° on H2O (about 1 m). Under the protocol of Ours (long), our student achieves the following results (with the ground-truth trajectory results included as a reference):

SplitEgo trajectoryS-PSNR ↑S-MSSIM ↑S-LPIPS ↓U-CLIP-I ↑[email protected] ↑MC ↑PSNR ↑SSIM ↑Loc Err ↓IoU ↑FVD ↓TF ↑MS ↑FPS ↑
New ActionDataset ground truth18.160.5880.3650.89347.59%53.48%17.470.5880.1180.447135.620.9810.99124.3
New ActionEstimated from exo video16.110.4690.4910.86530.98%36.47%15.260.4500.1640.308343.550.9730.98224.3
UnseenDataset ground truth17.880.5690.3940.89037.22%41.70%16.840.5370.1190.427144.330.9880.99424.3
UnseenEstimated from exo video15.280.4670.5170.84823.76%31.32%14.700.4430.1830.290368.420.9720.97924.3
H2ODataset ground truth23.210.7130.1180.92981.80%89.71%21.050.6800.0560.81494.170.9930.99424.3
H2OEstimated from exo video20.280.6680.1790.87163.50%71.26%18.560.6150.1150.744174.010.9750.98424.3

Although the estimated poses are less accurate than the ground truth trajectories, our student remains competitive with the baseline methods in Table 1, which all use the ground truth trajectories. Our results can further improve as head tracking becomes more accurate. We have added this experiment to the main paper as Sec. 4.4 (L502–528).

Comments on the paper

2 comments

We thank the reviewers for their insightful comments and acknowledge our highly relevant and novel streaming Exo2Ego setting (wAHK), the first method for this task (wAHK, 9ux3), novel (jYXg) and clever (9ux3) mechanisms well suited to Exo2Ego (jYXg, 9ux3), strong performance (wAHK), clear writing (jYXg, 9ux3), comprehensive experiments and ablations (9ux3), and a detailed appendix facilitating reproducibility (9ux3). We address the comments of each reviewer in the individual responses below.

Area_Chair_RJFPOpenReview ↗

Discussion with the authors

Dear reviewers,

the authors have provided responses for your reviews. Can you please let them know if your comments have been addressed?

Thank you, Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.