acceptodds
Under review as a conference paper at ICLR 2027

LEGO: A Lifting-Free Approach for Exocentric-to-Egocentric Video Generation

Abstract

Generating an egocentric video from a single exocentric recording is a challenging case of novel view synthesis, as the two cameras share little overlap and much of the target view is unobserved. Current state-of-the-art methods reconstruct the scene explicitly by estimating depth, lifting the video into a point cloud, and re-rendering it from the egocentric camera to condition a video diffusion model. This deterministic mapping assigns each pixel to a single reprojected location, which preserves texture but translates depth errors into misplaced content. We ask what a video diffusion model should receive as its condition and propose a lifting-free answer: a learned view synthesizer, an LVSM-style transformer fine-tuned to render the egocentric view directly without depth, point clouds, or reprojection, resolving cross-view correspondence internally. In contrast, its probabilistic mapping averages each region over candidate source locations according to a learned correspondence distribution, preserving structure while fine texture is averaged away. We argue that this trade-off suits a diffusion generator, whose denoising training excels at restoring detail, so an effective condition should prioritize structural alignment over sharpness. This distribution's concentration also yields a per-region confidence, used both to mask low-confidence regions and to guide the generator toward high-confidence areas during early layout-forming denoising steps. Our approach consistently outperforms the state-of-the-art explicit pipeline and generalizes to other datasets without retraining. The synthesizer thus supplies view structure, and the diffusion model its detail.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Exocentric-to-Egocentric Video Generation, Egocentric View Synthesis, Cross-View Correspondence

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_2FT8OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies exocentric-to-egocentric video generation given a calibrated exocentric video and a target egocentric camera trajectory. It replaces the depth estimation and point-cloud rendering stage of EgoX with an LVSM-style view synthesizer fine-tuned on paired ego-exo frames. Attention concentration provides a confidence score used to mask the rendered condition and to weight an inference-time correction, Adaptive Rendering Consistency (ARC), to the diffusion model’s clean latent estimate. Experiments report improvements over EgoX on seen and unseen Ego-Exo4D scenes, together with transfer to EgoHumans and Nymeria without additional training.

Critical strengths
  1. The learned rendering condition produces meaningful improvements on a difficult viewpoint transformation task. Against the released EgoX checkpoint evaluated with the same scoring code, unseen-scene PSNR improves from 13.53 to 15.91 dB and LPIPS decreases from 0.542 to 0.504. The transfer results on two additional datasets make the empirical contribution more convincing than a gain confined to familiar environments. Table 2 also shows that improvements are already present before adding ARC.

  2. The approach offers a useful simplification of the conditioning pipeline. It avoids depth estimation and explicit reconstruction while retaining the target camera specification. The reported condition construction time drops from 69.2 to 4.7 seconds per clip, and end-to-end generation decreases from 666.7 to 577.9 seconds. Extracting a reliability signal from the synthesizer and reusing it during sampling is a practical design that could be useful beyond this particular benchmark.

Critical weaknesses
  1. Appendix A.3 states that the ARC duration and squared confidence weighting were selected on the unseen split, which is also used to report generalization performance. A development split could have been held out from the training data. This does not invalidate the gains without ARC or the cross-dataset results, but it makes the incremental improvement attributed to ARC less conclusive.

  2. TF, MS, and DD are computed on the full exocentric–egocentric canvas. Since the observed exocentric view occupies a substantial part of this canvas, these scores do not isolate the temporal quality of the generated egocentric video. Reporting them on the egocentric output alone would provide a more relevant assessment.

  3. Averaged attention weights in a multilayer transformer do not necessarily represent physical source correspondences, since the source tokens have already mixed information across locations. Moreover, restricting the softmax to exocentric keys discards the total attention mass assigned to the source. Appendix C.4 shows a useful correlation between confidence and render error, but this supports the score as a reliability heuristic rather than establishing the correspondence interpretation in Eq. 3.

  4. The point-cloud variant uses a binary hole mask, whereas LEGO uses continuous squared confidence. The comparison therefore changes both where the correction is applied and how strongly it is applied. The poor performance of point-cloud ARC cannot be attributed to render misalignment alone. Constant or shuffled confidence controls on the same synthesizer render, and a strength-matched point-cloud variant, would help establish the benefit of the proposed confidence weighting.

  5. The learned and point-cloud renders differ in coverage, appearance, and task-specific supervision, in addition to alignment and sharpness. Tables 2 and 3 support the usefulness of the learned condition, but they do not establish which of these differences drives the improvement. A controlled comparison that varies blur and spatial misalignment separately would provide stronger evidence for the paper’s central explanation.

Questions for authors
  1. Further details on the ARC configurations explored on the unseen split and the criteria used to select the final setting would help assess parameter sensitivity. An evaluation with parameters selected on held-out training scenes would strengthen the generalization evidence.
  2. Comparisons with constant or shuffled confidence weights at a similar correction strength would help clarify the contribution of spatially varying confidence. A strength-matched point-cloud ARC variant would also make the comparison easier to interpret.
  3. Temporal metrics computed on the egocentric output alone, together with image reconstruction metrics within the common valid fisheye region, would help distinguish improvements in scene content and motion from differences in reproducing the image boundary.
  4. Clarification on whether the full EgoX configuration—point-cloud condition, exocentric tokens, and GGA—was evaluated under the same 20k-step training recipe would
5 comments
AuthorsOpenReview ↗

Response to Reviewer 2FT8 (1/3)

Thank you for the detailed review and the helpful comments. We have revised the paper throughout, and the new version differs considerably from the submission. We rewrote the text to make it clearer and more direct, and we added new experiments that we believe strengthen the paper, including those requested in the reviews. Since the changes are extensive, they are not marked individually, and each answer below points to the relevant section or table.

W1 / Q1. ARC settings were selected on the unseen split.

Correct: the unseen split is a test split, and the appendix now says so. ARC has two settings, the duration hh and the exponent of the confidence weight. We picked them with a coarse sweep on that split, without a formal criterion, and did not tune them further.

The table shows what this choice amounts to. There is no single best setting: both settings move the output along one trade-off. A longer or stronger correction (a lower exponent gives a larger weight) pulls the output closer to the render, which gains accuracy and costs sharpness. For the held-out evaluation you suggest, our models are trained on the full training split, so we used the closest set that needs no retraining: 100 clips of the seen split, which are not in the training set but come from training scenes. The trend is the same there.

Weight, hhSeen, 100 clips: PSNR / LPIPS / Sharpness / ChromaUnseen: PSNR / LPIPS / Sharpness / Chroma
No ARC18.95 / 0.339 / 0.96 / 0.8514.98 / 0.515 / 1.04 / 0.59
c2c^2, 0.219.18 / 0.332 / 0.92 / 0.8415.19 / 0.513 / 1.00 / 0.58
c2c^2, 0.419.47 / 0.325 / 0.88 / 0.8515.41 / 0.510 / 0.95 / 0.56
c2c^2, 0.619.83 / 0.316 / 0.83 / 0.8615.63 / 0.506 / 0.89 / 0.55
c2c^2, 0.8 (paper)20.30 / 0.307 / 0.77 / 0.8715.91 / 0.504 / 0.81 / 0.53
c2c^2, 1.021.33 / 0.331 / 0.60 / 0.9116.55 / 0.523 / 0.60 / 0.50
c1c^1, 0.821.29 / 0.307 / 0.65 / 0.8816.70 / 0.511 / 0.61 / 0.49
c3c^3, 0.819.70 / 0.317 / 0.85 / 0.8615.44 / 0.508 / 0.92 / 0.56
c4c^4, 0.819.37 / 0.325 / 0.89 / 0.8515.21 / 0.510 / 0.98 / 0.57

As you note, this concerns only the increment of ARC. In Table 2 ARC improves every metric on both splits. Its setting only decides where on the trade-off the output lies. EgoHumans and Nymeria use the same setting without re-selection.

W2 / Q3. TF, MS and DD on the egocentric output alone. Image metrics inside the valid fisheye region.

Table 1 now computes TF, MS and DD on the generated egocentric view only, for every method and dataset. The full-canvas values remain only in the appendix table that follows the protocol of EgoX.

Table 1 keeps the image metrics over the full frame. Inside the valid fisheye region (one mask per dataset, about 94% of the frame, the same for every method) they are, as PSNR / SSIM / LPIPS:

MethodSeenUnseenEgoHumansNymeria
Exo2Ego-V14.92 / 0.552 / 0.58613.63 / 0.451 / 0.65514.14 / 0.524 / 0.677Needs four views
TrajectoryCrafter12.98 / 0.359 / 0.64912.59 / 0.301 / 0.67013.46 / 0.373 / 0.63013.87 / 0.403 / 0.620
Wan Fun Control12.88 / 0.430 / 0.63512.27 / 0.350 / 0.67212.08 / 0.406 / 0.66812.39 / 0.402 / 0.682
Wan VACE13.56 / 0.480 / 0.61312.78 / 0.394 / 0.66813.62 / 0.487 / 0.63113.25 / 0.464 / 0.668
EgoX, retrained15.21 / 0.543 / 0.50213.30 / 0.424 / 0.57913.55 / 0.455 / 0.6189.43 / 0.386 / 0.717
LEGO20.01 / 0.676 / 0.32515.64 / 0.483 / 0.52815.22 / 0.520 / 0.57514.93 / 0.513 / 0.592

The ranking does not change: LEGO is best on every dataset, except for SSIM on EgoHumans, where Exo2Ego-V is marginally higher. Every method loses a little PSNR, because the exterior, which all methods reproduce, no longer counts.

AuthorsOpenReview ↗

Response to Reviewer 2FT8 (2/3)

W3. Averaged attention is not a physical correspondence. Restricting the softmax to exocentric keys discards attention mass.

Both points are fair, and the paper changed. The correspondence distribution should not be interpreted as a physical correspondence, because the exocentric tokens in these layers already contain information from multiple source locations. Section 3 now states that we use its concentration only as a heuristic for the reliability of the render. One sentence also overstated Fig. 5. On unseen scenes the advantage holds for the most confident windows only, and Section 4 now reports it for the 10% most confident windows.

The restriction of the softmax is deliberate. The confidence measures how concentrated the attention on the source is, not how much attention the source receives. Without the restriction the confidence ranks the windows almost identically, and the gate keeps largely the same windows.

W4 / Q2. Constant or shuffled confidence at the same strength. A strength-matched point-cloud ARC.

We ran both controls on the unseen split. The spatial pattern of the confidence does not contribute to ARC, and the point-cloud row did poorly because of its strength, as you suspected. What matters for ARC is the render it pulls toward, not how the weight varies across locations.

Every row is LEGO with the same generator and condition, and only ARC changes at inference. The mean weight over the latent grid is in parentheses.

ARC targetWeight (mean)Unseen: PSNR / LPIPS / FVD
No ARC14.98 / 0.515 / 415.08
Point-cloud renderHole mask, as submitted (0.536)13.72 / 0.618 / 812.88
Point-cloud renderVGGT-Omega confidence, rescaled (0.142)15.20 / 0.536 / 450.61
Synthesizer renderc2c^2, LEGO (0.142)15.91 / 0.504 / 404.77
Synthesizer renderc2c^2 shuffled within each frame (0.142)15.97 / 0.502 / 397.06
Synthesizer renderc2c^2 shuffled at every step (0.142)15.98 / 0.502 / 400.47

Shuffling the confidence changes nothing. We permuted the c2c^2 map of each latent frame over its locations, once per clip or again at every denoising step, so the mean weight stays the same and the pattern is gone. PSNR, LPIPS and FVD stay at the level of c2c^2. The gain of ARC therefore depends on its overall strength, which moves the output along the trade-off shown in W1, and not on where the weight is placed. The c2c^2 weight was a design choice, and we removed the claim that ARC needs a confidence map.

The point-cloud row did poorly because of its strength: the hole mask is several times as strong as c2c^2. A matched variant needs a confidence, which the point cloud of EgoX does not have, so we rebuilt it with VGGT-Omega. It predicts a depth and a confidence for every exocentric pixel. We scale the depth per clip with the metric depth that EgoX uses (ViPE) and render both into the egocentric view. The weight is this confidence minus its minimum, zero in the holes, scaled so that its mean equals that of c2c^2. At this strength the point cloud no longer collapses, but it gains much less PSNR than the synthesizer render, and LPIPS and FVD do not improve over no ARC.

The measured effect of the confidence is in the gate: in Table 2 gating lowers FVD on both splits. ARC reuses the same confidence as its weight, and all tables keep the submitted c2c^2. The point-cloud ARC row of the submission is no longer in the paper, and we report these controls here instead.

AuthorsOpenReview ↗

Response to Reviewer 2FT8 (3/3)

W5. The two renders also differ in coverage, appearance and supervision. Vary blur and misalignment separately.

Tables 2 and 4 do not separate these factors. We know no principled way to put blur and displacement on one scale, so we approached the question through what each render taught the generator. The two generators of Table 2 that are trained on the point-cloud render and on the synthesizer render (no gate, no exocentric tokens) differ only in their condition. At inference, on the unseen split, we replace it with the full ground-truth frames, the same input for both generators. A generator that has learned to follow its condition should then improve by a large margin.

Generator trained onCondition at inferenceUnseen: PSNR / LPIPS / FVD
Point-cloud renderOwn render (Table 2)13.42 / 0.595 / 549.77
Ground truth14.78 / 0.508 / 565.41
Synthesizer renderOwn render (Table 2)14.35 / 0.567 / 581.90
Ground truth20.64 / 0.236 / 283.61

The generator trained on the blurry, better aligned render improves by a large margin with the ground truth, on every clip and in every metric. The generator trained on the sharp, misaligned render improves far less with the same frames, and its FVD does not improve. The untuned synthesizer render is a second control. It has the coverage and appearance of the fine-tuned one and is slightly sharper, but it has the lowest ZNCC of the three (Table 4) and conditions the generator no better than the point cloud (Table 2).

The test is indirect. Varying blur and misalignment inside the training data would need retraining on modified renders, which we did not do. The analysis is now in Section 4 (Table 5), and the introduction states the claim as a hypothesis that the experiments test.

Q4. Was the full EgoX configuration evaluated under the same 20k-step recipe?

Not in the submission. We retrained it (point-cloud render, exocentric tokens, GGA) with the same training recipe as LEGO: same backbone, LoRA rank, learning rate, 20k steps, 4 GPUs. Table 1 now reports this model as its EgoX row on all four test sets.

Seen: PSNR / LPIPS / FVDUnseen: PSNR / LPIPS / FVD
EgoX†, released checkpoint15.57 / 0.466 / 197.9513.53 / 0.542 / 455.06
EgoX, retrained (Table 1)15.48 / 0.479 / 195.3113.57 / 0.553 / 490.44
LEGO20.28 / 0.310 / 155.5615.91 / 0.504 / 404.77

The retrained EgoX lands where the released checkpoint does, so the comparison does not depend on the recipe. At equal steps EgoX is also the slower model to train: the GGA bias is added to the attention scores, which rules out the FlashAttention kernel (timings in the appendix).

Reviewer_653KOpenReview ↗
Rating
3(+2)
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper does exocentric-to-egocentric video reframing.

The author selected an implicit LVSM-style approach, in contrast to explicit point cloud re-rendering recipe ("lifting"), to implement reframing. It is a regular conditioned video diffusion recipe. The denoising scheme is [noise ego latent | clean exo latent, synthesized ego latent] -> [clean ego latent].

To differentiate from prior work that uses point clouds (EgoX), the authors trained a LVSM-style view synthesizer to initialize the exo-to-ego warping. The confidence map is provided to the diffusion model to better control the conditioning.

The view synthesizer is finetuned from LVSM on Ego-Exo4D. The diffusion model is finetuned on Ego-Exo4D. The end2end pipeline is evaluated on Ego-Exo4D and zero-shot evaluated on EgoHumans and Nymeria.

The evaluation results are consistently better than baselines.

Critical strengths

The paper explored the combination of LVSM and video diffusion model, which is quite constructive for 3D-free conditioning.

Critical weaknesses
  • Major experiment Table 1 is unfair. The view synthesizer is finetuned on the same Ego-Exo4D benchmark, meaning the input condition to the diffusion model is much stronger than baselines (with all the priors). So the task becomes too easy for the diffusion model. Table 4 does compensate it a bit on zero-shot evaluation sets. I prefer extending Table 4 as the main table with all the baselines.
  • Gated conditioning doesn't differentiate the implicit vs explicit methods and is not a novelty. For instance, VGGT-Omega produces confidence mask and can be directly applied to point cloud based approaches.
  • In Eq (2), L is not defined anywhere in the main paper. I assume it is 8 in the appendix but there is no clear association.
  • AI slop is pervasive in the paper. Here are some blatant examples:
  1. L87: Alignment first, detail second also describes the sampling trajectory itself: layout is decided in the early, high-noise steps and detail emerges later, so structural guidance is most useful early.
  2. L131: We rely on the same effect in a synthesizer fine-tuned for the exocentric-to-egocentric view change: its render serves as the generator’s condition, and the concentration of its correspondence distribution as a per-region confidence
  3. L215: Two kinds of error. The two mappings fail differently.
  4. L239: The published LVSM checkpoint, which we call the un- tuned synthesizer, is trained to interpolate between nearby pinhole views, whereas our target is an extrapolation to a fisheye headset camera that shares little overlap with the source.
  5. Table 2: Arm: Conditioning input: what fills it
  6. L478: LEGO conditions a video diffusion model on the render of a learned view synthesizer, not a point- cloud reconstruction, on one criterion: a condition should be aligned before it is sharp, because the generator restores detail far more readily than it corrects misplaced content.
Questions for authors
  • As a reviewer, I respect the authors and did not use any AI tools. Out of mutual respect, I would like to see a revised paper without AI slop. My original score for technical contribution is 2. If the authors are able to de-slop properly, 2 will be my starting score.
  • I would like to see a promoted Table 4 with baselines in Table 1.
  • I would like to see an ablation on EgoX with backbones that supports probabilistic conditioning (e.g. VGGT-Omega).
5 comments
AuthorsOpenReview ↗

Response to Reviewer 653K (1/3)

Thank you for the review and the helpful comments. We have revised the paper throughout, and the new version differs considerably from the submission. We rewrote the text to make it clearer and more direct, and we added new experiments that we believe strengthen the paper, including those requested in the reviews. Since the changes are extensive, they are not marked individually, and each answer below points to the relevant section or table.

W1 / Q2. Table 1 is unfair because the synthesizer is fine-tuned on Ego-Exo4D. Promote Table 4 with all baselines.

Table 4 is now part of Table 1. Every baseline is run by us and evaluated on EgoHumans and Nymeria, which no component of LEGO has seen, with the same evaluation code (appendix, "Baselines").

EgoHumans: PSNR / LPIPS / FVDNymeria: PSNR / LPIPS / FVD
Exo2Ego-V (four exocentric views)14.43 / 0.642 / 1011.04Needs four views
TrajectoryCrafter13.72 / 0.617 / 965.7114.15 / 0.605 / 447.13
Wan Fun Control12.38 / 0.635 / 897.6412.69 / 0.647 / 374.55
Wan VACE13.92 / 0.599 / 960.9213.55 / 0.633 / 421.76
EgoX†, released checkpoint13.74 / 0.573 / 421.8311.64 / 0.637 / 296.98
EgoX, retrained (Table 1)13.84 / 0.586 / 425.939.73 / 0.683 / 329.51
LEGO15.52 / 0.545 / 332.4115.23 / 0.561 / 210.27

LEGO is the best method on both zero-shot sets, on PSNR, SSIM, LPIPS, CLIP-I and FVD. EgoX in Table 1 is retrained by us with our training recipe, and EgoX† is the released checkpoint (appendix). LEGO and EgoX use the same training data, the training split of Ego-Exo4D and nothing else. Both fine-tune the generator on it, and we also fine-tune the synthesizer. The weights trained this way are the ones evaluated zero-shot above.

To test whether the result depends on the synthesizer having seen Ego-Exo4D, we replaced only the synthesizer. We trained it with the same recipe and the same amount of training data, but on EgoHumans and Nymeria instead of Ego-Exo4D. We then froze it and retrained the generator on Ego-Exo4D with its renders, exactly as for LEGO (Section 4, Table 6).

MethodSynthesizer trained onPSNRSSIMLPIPSCLIP-IFVDTFMSDD
EgoX, retrained (Table 1)– (point cloud)13.570.4480.5530.894490.440.9690.9890.990
LEGOEgoHumans + Nymeria15.750.4990.5120.893424.140.9760.9911.000
LEGOEgo-Exo4D15.910.5050.5040.897404.770.9770.9911.000

With a synthesizer that has never seen Ego-Exo4D, the result on the unseen split is nearly unchanged and stays well above EgoX. The gain of LEGO does not come from fine-tuning the synthesizer on the benchmark.

AuthorsOpenReview ↗

Response to Reviewer 653K (2/3)

W2 / Q3. Gating is not what separates implicit from explicit methods. Ablation on EgoX with VGGT-Omega.

We agree and do not claim gating as a novelty: the condition of EgoX can be made probabilistic in the same way. It needs a model that predicts a confidence, so we use VGGT-Omega. Its depth is relative, so we scale it with the metric depth of ViPE, rebuild the point-cloud render of EgoX from it, and gate the render by the VGGT-Omega confidence as in our method. The threshold is 7.5, the value that acts like our τ=0.3\tau{=}0.3. Section 4 now reports this in Table 3.

ModelConditionUnseen: PSNR / LPIPS / FVD
EgoXPoint cloud13.57 / 0.553 / 490.44
EgoX (VGGT-Omega)Gated point cloud14.61 / 0.557 / 420.80
LEGOGated synthesizer render15.91 / 0.504 / 404.77

EgoX in the first row is the retrained model of Table 1. With the gated VGGT-Omega point cloud, EgoX improves, but LEGO remains ahead.

AuthorsOpenReview ↗

Response to Reviewer 653K (3/3)

W3. LL in Eq. 2 is not defined.

Section 3 now defines it where the equation is introduced: L=8L{=}8 of the 24 layers of the synthesizer.

W4 / Q1. Writing.

We apologize. We used a large language model to draft and edit text, as disclosed in the submission, and the result reads the way you describe. That is our responsibility. The method, the experiments and the analysis are our own. We rewrote the passages you listed and revised the writing of the whole paper.

L87 in the introduction now reads "The scene layout is largely established during the early denoising steps at high-noise levels." L131 in the related work now reads "We fine-tune a view synthesizer for the exocentric-to-egocentric view change and use its render to condition the generator. We estimate confidence for each target region from the concentration of its correspondence distribution over source locations (Section 3.2)."

At L215 in the method, the heading is now "Explicit and learned mappings", and "The two mappings fail differently." is deleted. The paragraph now starts with "The explicit mapping preserves texture by assigning a single source pixel to each target pixel. However, errors in the estimated depth can cause the wrong source pixel to be selected." L239 now reads "The released LVSM model was trained to interpolate between nearby pinhole views, whereas our task requires generating a fisheye view with little overlap with the exocentric input. We refer to the model before fine-tuning as the untuned synthesizer."

In Table 2, the column "Arm: Conditioning input: what fills it" is gone, and each row lists its components as columns (Condition, Gating, Exo., GGA, ARC). L478 in the conclusion now reads "LEGO conditions a video diffusion model on the render of a learned view synthesizer instead of a point-cloud render. Our experiments indicate that the spatial alignment of the condition matters more than sharpness, because the generator can recover detail but cannot correct misplaced structures."

Reviewer_SUPfOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper is on exocentric-to-egocentric video generation. From a single third-person video, calibrated exocentric/egocentric cameras, and an egocenetric trajectory, the system synthesizes the egocentric video.

The paper closely follows the setting of the work EgoX, and changes the core explicit point cloud and reprojection part. Instead, they render directly the egocentric frame.

The paper proposes a per-location confidence off the synthesizer's attention for gating low-confidence regions of the condition to gray, and for the prpoposed Adaptive Rendering Consistency (ARC) approach. ARC is a training-free blend of the flow-matching clean estimate toward the encoded render, similar to a soft-mask inpainting guidance. This replaces EgoX's trained geometry-guided attention (GGA).

The authors present results both on Ego-Exo4D, which they use for training and on EgoHumans and Nymeria datasets which were not used for training. The ablations compare different conditioning inputs and guidance options, and an analysis of the renders supports the idea that alignment matters more than sharpness.

Critical strengths

I beleive that the main strength is clear presentation of contributions and numbers. It clearly follows EgoX and and shows improvement over it in a relevant problem. Spcifically, measure how well aligned and how sharp each conditioning render is (something they claim is important) and show show that the blurrier but better aligned render works better as a condition under the same training formulation. The untuned synthesizer is a useful control here. It is sharper and covers almost the whole frame, yet it does no better than the point-cloud render, which shows that the gain comes from adapting the synthesizer to the egocentric camera rather than from the architecture itself. The “aligned before sharp” insight could apply to other problems with large viewpoint changes.

The second strength is that the method is simpler and cheaper than EgoX and still gives large gains. It removes depth estimation and point-cloud rendering, so building the condition takes about 5 seconds per clip instead of about 70. The results on EgoHumans and Nymeria are also convincing: the same weights and settings are used without retraining, and the gains hold on both datasets. This is the strongest sign that the improvement is not just a result of tuning on Ego-Exo4D.

Critical weaknesses

One potential issue is that in Table 1 is against the released EgoX checkpoint, which was trained with roughly half the compute of LEGO. Table 2 is meant to control for this, but it is missing the most important row: EgoX itself retrained under the authors’ recipe, with the point-cloud render, the exocentric tokens and GGA. The point-cloud condition is only tested with the exocentric tokens hidden, and all other components are only added on top of the synthesizer render. That hidden setting is also the one most favorable to the paper’s argument, since the model cannot look at the source view to correct misplaced content. Under the same recipe, the point-cloud model with hidden tokens already matches the released EgoX, so a full EgoX trained this way could be noticeably stronger.

Questions for authors

It is not clear where the large PSNR gain comes from. The paper shows that outputs conditioned on the synthesizer render are about as sharp as the ground truth, but this check does not seem to include the final model with ARC, which pulls toward the blurry render for most of the denoising steps. The appendix also shows that the outputs inherit much of the render’s color loss, and desaturated, averaged colors can raise PSNR on their own. LPIPS and FVD also improve, so part of the gain is clearly real, but it would help to report sharpness and color for the final model with and without ARC, and PSNR computed on luminance only, to separate better alignment from averaging.

The paper states ARC settings were chosen on the unseen split, is this the test split? The zero-shot results on EgoHumans and Nymeria address this but still would be good to make this clear.

5 comments
AuthorsOpenReview ↗

Response to Reviewer SUPf (1/2)

Thank you for the review and the helpful comments. We have revised the paper throughout, and the new version differs considerably from the submission. We rewrote the text to make it clearer and more direct, and we added new experiments that we believe strengthen the paper, including those requested in the reviews. Since the changes are extensive, they are not marked individually, and each answer below points to the relevant section or table.

W1. The released EgoX had about half the compute, and EgoX retrained under our recipe is missing.

We retrained EgoX (point-cloud render, exocentric tokens, GGA) with the same training recipe as LEGO: same backbone, LoRA rank, learning rate, 20k steps, 4 GPUs. Table 1 now reports this model as its EgoX row on all four test sets.

Seen: PSNR / LPIPS / FVDUnseen: PSNR / LPIPS / FVD
EgoX†, released checkpoint15.57 / 0.466 / 197.9513.53 / 0.542 / 455.06
EgoX, retrained (Table 1)15.48 / 0.479 / 195.3113.57 / 0.553 / 490.44
LEGO20.28 / 0.310 / 155.5615.91 / 0.504 / 404.77

The retrained EgoX lands where the released checkpoint does, so the training budget does not explain the gap to LEGO. At equal steps EgoX is also the slower model to train: the GGA bias is added to the attention scores, which rules out the FlashAttention kernel (timings in the appendix).

Q1. Where does the PSNR gain come from? Sharpness and color with and without ARC. PSNR on luminance.

The submission reported sharpness and color for a few outputs on the unseen split only. We now measure them with one protocol (Appendix B.3) on all clips of both test splits, for the two renders and for the generators conditioned on them. Each output row is a generator of Table 2, and ARC is applied at inference. Values are relative to the ground truth (1 = the recording). For the point-cloud render they are measured on its rendered pixels. Sharpness is now the mean absolute Laplacian response, which is less dominated by a few high-contrast clips than the variance used in the submission. Table 4 uses the same measure. The last two rows are the final model without and with ARC.

Seen: Sharpness / ChromaUnseen: Sharpness / Chroma
Point-cloud render1.35 / 0.871.13 / 0.70
Synthesizer render0.77 / 0.940.67 / 0.48
Output, point-cloud render1.06 / 1.141.00 / 0.82
Output, synthesizer render0.92 / 0.770.91 / 0.45
+ gating1.00 / 0.961.04 / 0.61
+ exocentric tokens0.99 / 0.861.04 / 0.59
+ ARC (LEGO)0.81 / 0.880.81 / 0.53

Output sharpness is not set by the sharpness of the condition. The synthesizer render is much softer than the point-cloud render, but the outputs conditioned on the two are close in sharpness, and with the gate the output is as sharp as the recording.

ARC is a trade-off, which the submission did not show (Section 4 now does). It pulls the output toward the render, which is soft, so sharpness falls while every metric of Table 2 improves. Smoothing alone does not give this gain: blurring the output without ARC to the same sharpness leaves PSNR almost unchanged (Appendix C.2).

Color follows the condition. On unseen scenes the synthesizer render is desaturated, and so is the output conditioned on it, unlike the output conditioned on the point-cloud render. The gate recovers part of the color.

Most of the PSNR gain over EgoX already appears before ARC is added (Table 2), while the output is still as sharp as the recording. PSNR and SSIM in the paper were already computed on luminance (the appendix now says so), so a loss of color cannot raise them.

AuthorsOpenReview ↗

Response to Reviewer SUPf (2/2)

Q2. Were the ARC settings chosen on the test split?

Yes. The unseen split is one of the two test splits, and the appendix now says so. ARC has two settings, the duration hh and the exponent of the confidence weight. We picked them with a coarse sweep on that split and did not tune them further.

The table shows what this choice amounts to. There is no single best setting: both settings move the output along one trade-off. A longer or stronger correction (a lower exponent gives a larger weight) pulls the output closer to the render, which gains accuracy and costs sharpness. The trend is the same on 100 clips of the seen split, which are not in the training set but come from training scenes.

Weight, hhSeen, 100 clips: PSNR / LPIPS / Sharpness / ChromaUnseen: PSNR / LPIPS / Sharpness / Chroma
No ARC18.95 / 0.339 / 0.96 / 0.8514.98 / 0.515 / 1.04 / 0.59
c2c^2, 0.219.18 / 0.332 / 0.92 / 0.8415.19 / 0.513 / 1.00 / 0.58
c2c^2, 0.419.47 / 0.325 / 0.88 / 0.8515.41 / 0.510 / 0.95 / 0.56
c2c^2, 0.619.83 / 0.316 / 0.83 / 0.8615.63 / 0.506 / 0.89 / 0.55
c2c^2, 0.8 (paper)20.30 / 0.307 / 0.77 / 0.8715.91 / 0.504 / 0.81 / 0.53
c2c^2, 1.021.33 / 0.331 / 0.60 / 0.9116.55 / 0.523 / 0.60 / 0.50
c1c^1, 0.821.29 / 0.307 / 0.65 / 0.8816.70 / 0.511 / 0.61 / 0.49
c3c^3, 0.819.70 / 0.317 / 0.85 / 0.8615.44 / 0.508 / 0.92 / 0.56
c4c^4, 0.819.37 / 0.325 / 0.89 / 0.8515.21 / 0.510 / 0.98 / 0.57

What is clear is the benefit of ARC itself: in Table 2 it improves every metric on both splits. Its setting only decides where on the trade-off the output lies, and we use h=0.8h{=}0.8 with c2c^2. EgoHumans and Nymeria use this setting without re-selection.

Dear Reviewer SUPf,

Thank you again for your thoughtful review. As the discussion draws to a close, we wanted to follow up on your suggestions.

After retraining the full EgoX model with LEGO’s training recipe, LEGO still outperforms EgoX on the unseen split in PSNR, LPIPS, and FVD.

We also report sharpness and chroma with and without ARC. Most of the luminance PSNR gain is already present before ARC, and blurring the output to match ARC’s sharpness barely changes PSNR. Our sweep showed no single best setting across metrics. On both splits, stronger or longer correction increased PSNR at the cost of sharpness.

We hope this addresses your concerns and would be happy to clarify any remaining points.

Best regards, The Authors

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response (to all reviewers)

We thank all reviewers for their careful reading and constructive feedback. We are glad that the reviewers found the combination of a learned view synthesizer and a video diffusion model a constructive direction for 3D-free conditioning (653K), the improvements over EgoX meaningful (2FT8), the analysis of alignment and sharpness insightful (SUPf), the pipeline simpler and cheaper than EgoX (SUPf, 2FT8), and the results on EgoHumans and Nymeria without retraining convincing (SUPf, 2FT8).

Addressing these comments has substantially improved the quality of the paper, and we are sincerely grateful for them. We revised the paper throughout, rewrote the text to be clearer, and added the experiments the reviewers asked for. Because the changes are extensive, they are not marked in color. The main changes are:

  • Baselines (Table 1). EgoX is retrained with the same recipe as LEGO, every baseline is run by us on all three datasets, and TF, MS and DD are computed on the egocentric view (SUPf, 653K, 2FT8).

  • New experiments (Section 4, Tables 3, 5 and 6). A synthesizer trained without Ego-Exo4D, confidence gating applied to the point-cloud condition of EgoX, and the ground truth given as the condition to show what each render teaches the generator (653K, 2FT8).

  • Analysis and clarifications (Section 3, Appendices A.3, B.3 and C.2). Sharpness and color with and without ARC, the ARC settings and the test split, the role of the confidence, and the definition of LL (SUPf, 653K, 2FT8).

  • Writing (whole paper). The text is rewritten throughout, including the passages listed by Reviewer 653K (653K).

Each individual response gives the details and points to the relevant sections and tables.

We thank Reviewers SUPf, 653K, and 2FT8 for their constructive feedback and for evaluating the additional experiments. We also thank the Area Chair and Senior Area Chair for coordinating the review process.

The reviewers highlighted the promise of conditioning video diffusion on a learned view synthesizer’s output (653K), the broader relevance of our findings on alignment and sharpness (SUPf), and LEGO’s simpler pipeline and gains on EgoHumans and Nymeria without retraining (SUPf, 2FT8).

During the review process, their questions strengthened the paper by prompting a fairer comparison and a closer examination of how alignment and smoothing affect the results.

Following the discussion, Reviewer 653K raised their rating from 1 to 3, while Reviewers SUPf and 2FT8 maintained their positive ratings of 3.

The overall ratings thus moved from 3/1/3 to 3/3/3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.