- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies exocentric-to-egocentric video generation given a calibrated exocentric video and a target egocentric camera trajectory. It replaces the depth estimation and point-cloud rendering stage of EgoX with an LVSM-style view synthesizer fine-tuned on paired ego-exo frames. Attention concentration provides a confidence score used to mask the rendered condition and to weight an inference-time correction, Adaptive Rendering Consistency (ARC), to the diffusion model’s clean latent estimate. Experiments report improvements over EgoX on seen and unseen Ego-Exo4D scenes, together with transfer to EgoHumans and Nymeria without additional training.
Critical strengths
-
The learned rendering condition produces meaningful improvements on a difficult viewpoint transformation task. Against the released EgoX checkpoint evaluated with the same scoring code, unseen-scene PSNR improves from 13.53 to 15.91 dB and LPIPS decreases from 0.542 to 0.504. The transfer results on two additional datasets make the empirical contribution more convincing than a gain confined to familiar environments. Table 2 also shows that improvements are already present before adding ARC.
-
The approach offers a useful simplification of the conditioning pipeline. It avoids depth estimation and explicit reconstruction while retaining the target camera specification. The reported condition construction time drops from 69.2 to 4.7 seconds per clip, and end-to-end generation decreases from 666.7 to 577.9 seconds. Extracting a reliability signal from the synthesizer and reusing it during sampling is a practical design that could be useful beyond this particular benchmark.
Critical weaknesses
-
Appendix A.3 states that the ARC duration and squared confidence weighting were selected on the unseen split, which is also used to report generalization performance. A development split could have been held out from the training data. This does not invalidate the gains without ARC or the cross-dataset results, but it makes the incremental improvement attributed to ARC less conclusive.
-
TF, MS, and DD are computed on the full exocentric–egocentric canvas. Since the observed exocentric view occupies a substantial part of this canvas, these scores do not isolate the temporal quality of the generated egocentric video. Reporting them on the egocentric output alone would provide a more relevant assessment.
-
Averaged attention weights in a multilayer transformer do not necessarily represent physical source correspondences, since the source tokens have already mixed information across locations. Moreover, restricting the softmax to exocentric keys discards the total attention mass assigned to the source. Appendix C.4 shows a useful correlation between confidence and render error, but this supports the score as a reliability heuristic rather than establishing the correspondence interpretation in Eq. 3.
-
The point-cloud variant uses a binary hole mask, whereas LEGO uses continuous squared confidence. The comparison therefore changes both where the correction is applied and how strongly it is applied. The poor performance of point-cloud ARC cannot be attributed to render misalignment alone. Constant or shuffled confidence controls on the same synthesizer render, and a strength-matched point-cloud variant, would help establish the benefit of the proposed confidence weighting.
-
The learned and point-cloud renders differ in coverage, appearance, and task-specific supervision, in addition to alignment and sharpness. Tables 2 and 3 support the usefulness of the learned condition, but they do not establish which of these differences drives the improvement. A controlled comparison that varies blur and spatial misalignment separately would provide stronger evidence for the paper’s central explanation.
Questions for authors
- Further details on the ARC configurations explored on the unseen split and the criteria used to select the final setting would help assess parameter sensitivity. An evaluation with parameters selected on held-out training scenes would strengthen the generalization evidence.
- Comparisons with constant or shuffled confidence weights at a similar correction strength would help clarify the contribution of spatially varying confidence. A strength-matched point-cloud ARC variant would also make the comparison easier to interpret.
- Temporal metrics computed on the egocentric output alone, together with image reconstruction metrics within the common valid fisheye region, would help distinguish improvements in scene content and motion from differences in reproducing the image boundary.
- Clarification on whether the full EgoX configuration—point-cloud condition, exocentric tokens, and GGA—was evaluated under the same 20k-step training recipe would
5 comments
Response to Reviewer 2FT8 (2/3)
W3. Averaged attention is not a physical correspondence. Restricting the softmax to exocentric keys discards attention mass.
Both points are fair, and the paper changed. The correspondence distribution should not be interpreted as a physical correspondence, because the exocentric tokens in these layers already contain information from multiple source locations. Section 3 now states that we use its concentration only as a heuristic for the reliability of the render. One sentence also overstated Fig. 5. On unseen scenes the advantage holds for the most confident windows only, and Section 4 now reports it for the 10% most confident windows.
The restriction of the softmax is deliberate. The confidence measures how concentrated the attention on the source is, not how much attention the source receives. Without the restriction the confidence ranks the windows almost identically, and the gate keeps largely the same windows.
W4 / Q2. Constant or shuffled confidence at the same strength. A strength-matched point-cloud ARC.
We ran both controls on the unseen split. The spatial pattern of the confidence does not contribute to ARC, and the point-cloud row did poorly because of its strength, as you suspected. What matters for ARC is the render it pulls toward, not how the weight varies across locations.
Every row is LEGO with the same generator and condition, and only ARC changes at inference. The mean weight over the latent grid is in parentheses.
| ARC target | Weight (mean) | Unseen: PSNR / LPIPS / FVD |
|---|---|---|
| No ARC | 14.98 / 0.515 / 415.08 | |
| Point-cloud render | Hole mask, as submitted (0.536) | 13.72 / 0.618 / 812.88 |
| Point-cloud render | VGGT-Omega confidence, rescaled (0.142) | 15.20 / 0.536 / 450.61 |
| Synthesizer render | , LEGO (0.142) | 15.91 / 0.504 / 404.77 |
| Synthesizer render | shuffled within each frame (0.142) | 15.97 / 0.502 / 397.06 |
| Synthesizer render | shuffled at every step (0.142) | 15.98 / 0.502 / 400.47 |
Shuffling the confidence changes nothing. We permuted the map of each latent frame over its locations, once per clip or again at every denoising step, so the mean weight stays the same and the pattern is gone. PSNR, LPIPS and FVD stay at the level of . The gain of ARC therefore depends on its overall strength, which moves the output along the trade-off shown in W1, and not on where the weight is placed. The weight was a design choice, and we removed the claim that ARC needs a confidence map.
The point-cloud row did poorly because of its strength: the hole mask is several times as strong as . A matched variant needs a confidence, which the point cloud of EgoX does not have, so we rebuilt it with VGGT-Omega. It predicts a depth and a confidence for every exocentric pixel. We scale the depth per clip with the metric depth that EgoX uses (ViPE) and render both into the egocentric view. The weight is this confidence minus its minimum, zero in the holes, scaled so that its mean equals that of . At this strength the point cloud no longer collapses, but it gains much less PSNR than the synthesizer render, and LPIPS and FVD do not improve over no ARC.
The measured effect of the confidence is in the gate: in Table 2 gating lowers FVD on both splits. ARC reuses the same confidence as its weight, and all tables keep the submitted . The point-cloud ARC row of the submission is no longer in the paper, and we report these controls here instead.
Response to Reviewer 2FT8 (3/3)
W5. The two renders also differ in coverage, appearance and supervision. Vary blur and misalignment separately.
Tables 2 and 4 do not separate these factors. We know no principled way to put blur and displacement on one scale, so we approached the question through what each render taught the generator. The two generators of Table 2 that are trained on the point-cloud render and on the synthesizer render (no gate, no exocentric tokens) differ only in their condition. At inference, on the unseen split, we replace it with the full ground-truth frames, the same input for both generators. A generator that has learned to follow its condition should then improve by a large margin.
| Generator trained on | Condition at inference | Unseen: PSNR / LPIPS / FVD |
|---|---|---|
| Point-cloud render | Own render (Table 2) | 13.42 / 0.595 / 549.77 |
| Ground truth | 14.78 / 0.508 / 565.41 | |
| Synthesizer render | Own render (Table 2) | 14.35 / 0.567 / 581.90 |
| Ground truth | 20.64 / 0.236 / 283.61 |
The generator trained on the blurry, better aligned render improves by a large margin with the ground truth, on every clip and in every metric. The generator trained on the sharp, misaligned render improves far less with the same frames, and its FVD does not improve. The untuned synthesizer render is a second control. It has the coverage and appearance of the fine-tuned one and is slightly sharper, but it has the lowest ZNCC of the three (Table 4) and conditions the generator no better than the point cloud (Table 2).
The test is indirect. Varying blur and misalignment inside the training data would need retraining on modified renders, which we did not do. The analysis is now in Section 4 (Table 5), and the introduction states the claim as a hypothesis that the experiments test.
Q4. Was the full EgoX configuration evaluated under the same 20k-step recipe?
Not in the submission. We retrained it (point-cloud render, exocentric tokens, GGA) with the same training recipe as LEGO: same backbone, LoRA rank, learning rate, 20k steps, 4 GPUs. Table 1 now reports this model as its EgoX row on all four test sets.
| Seen: PSNR / LPIPS / FVD | Unseen: PSNR / LPIPS / FVD | |
|---|---|---|
| EgoX†, released checkpoint | 15.57 / 0.466 / 197.95 | 13.53 / 0.542 / 455.06 |
| EgoX, retrained (Table 1) | 15.48 / 0.479 / 195.31 | 13.57 / 0.553 / 490.44 |
| LEGO | 20.28 / 0.310 / 155.56 | 15.91 / 0.504 / 404.77 |
The retrained EgoX lands where the released checkpoint does, so the comparison does not depend on the recipe. At equal steps EgoX is also the slower model to train: the GGA bias is added to the attention scores, which rules out the FlashAttention kernel (timings in the appendix).
Response to Reviewer 2FT8 (1/3)
Thank you for the detailed review and the helpful comments. We have revised the paper throughout, and the new version differs considerably from the submission. We rewrote the text to make it clearer and more direct, and we added new experiments that we believe strengthen the paper, including those requested in the reviews. Since the changes are extensive, they are not marked individually, and each answer below points to the relevant section or table.
W1 / Q1. ARC settings were selected on the unseen split.
Correct: the unseen split is a test split, and the appendix now says so. ARC has two settings, the duration h and the exponent of the confidence weight. We picked them with a coarse sweep on that split, without a formal criterion, and did not tune them further.
The table shows what this choice amounts to. There is no single best setting: both settings move the output along one trade-off. A longer or stronger correction (a lower exponent gives a larger weight) pulls the output closer to the render, which gains accuracy and costs sharpness. For the held-out evaluation you suggest, our models are trained on the full training split, so we used the closest set that needs no retraining: 100 clips of the seen split, which are not in the training set but come from training scenes. The trend is the same there.
As you note, this concerns only the increment of ARC. In Table 2 ARC improves every metric on both splits. Its setting only decides where on the trade-off the output lies. EgoHumans and Nymeria use the same setting without re-selection.
W2 / Q3. TF, MS and DD on the egocentric output alone. Image metrics inside the valid fisheye region.
Table 1 now computes TF, MS and DD on the generated egocentric view only, for every method and dataset. The full-canvas values remain only in the appendix table that follows the protocol of EgoX.
Table 1 keeps the image metrics over the full frame. Inside the valid fisheye region (one mask per dataset, about 94% of the frame, the same for every method) they are, as PSNR / SSIM / LPIPS:
The ranking does not change: LEGO is best on every dataset, except for SSIM on EgoHumans, where Exo2Ego-V is marginally higher. Every method loses a little PSNR, because the exterior, which all methods reproduce, no longer counts.