acceptodds
Under review as a conference paper at ICLR 2027

GRAFT: Geometry-First Video Retaking via Appearance-First Training

Abstract

Video retaking aims to reproduce an observed dynamic scene along a new camera trajectory while preserving its content and motion. A central challenge is locating relevant source evidence across frames as viewpoint changes, motion, and occlusion alter visibility. We introduce GRAFT, a video retaking framework built around Addressing Distillation, which transfers source-addressing behavior from appearance synthesis to target-view geometry prediction. Our key insight is that appearance synthesis conditioned on known target geometry provides a task for learning where to retrieve source evidence. Through Appearance-First Training, we first train an appearance expert to synthesize target RGB video from source observations and ground-truth target depth. We then probe the frozen expert at the pure-noise endpoint under the same depth conditioning, extracting its target-to-source attention distributions as soft addressing patterns. These distributions supervise a geometry expert's source addressing alongside its depth generation objective, without constructing explicit geometric correspondence labels. At inference, Geometry-First Retaking reverses the order: the geometry expert predicts target-view metric depth, which guides the same appearance expert in synthesizing the retaken video. Evaluations on dynamic-scene and human-centric video retaking demonstrate improved camera trajectory following, geometric consistency, and subject motion preservation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video Generation, Video Retaking

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_p7V7OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

GRAFT breaks down video novel-view synthesis (NVS) (or video retaking) into two stages: Target depth synthesis (based on the source video and target camera trajectory) and appearance transfer (transferring the appearance of the source video onto some given depth video). During training, the appearance expert is first trained with source-target video pairs (along with the target depth). The geometry expert is then trained with addressing distillation, which guides it to mirror the source-target attention map of the trained and frozen appearance expert. During inference, the geometry expert is first run to obtain target depths before using the appearance expert to transfer the source video's appearance onto said target depth. Doing so removes the need to do per-frame 4D point cloud renders that can introduce artifacts from 4D reconstruction which explicit-prior methods can suffer from (e.g., TrajectoryCrafter), while improving camera control precision compared to implicit-prior methods (e.g., ReCamMaster).

Critical strengths
  1. Hints of generality of addressing distillation (AD): The authors show in the ablations that AD not only helps the two-stage framework of GRAFT but also helps when the camera condition is warped target-view depth (which on the surface seems to require less source-target correspondence as the model is partially 'inpainting' that target-view depth). This has the potential wider implication that training for source-target correspondence is just generally a good 'representation learning' objective that can be used to distill/supervise video NVS models which require this correspondence more explicitly.
  2. Two-stage pipeline being less explicit: The two-stage pipeline of GRAFT makes its approach potentially more scalable as it is not constrained by the quality of the point cloud render (from the 4D reconstructor) which many baselines do. Moreover, compared to implicit-prior methods (e.g., ReCamMaster, CamCloneMaster), explicit depth grounding with the depth anchor resolves the scale ambiguity of camera extrinsics/reference videos.
Critical weaknesses
  1. Missing baselines: The paper is missing several important baselines which have been shown to outperform existing baselines in the paper (ReCamMaster, TrajectoryCrafter, CamCloneMaster, and EX-4D). Namely, the paper is missing comparisons to Vista4D (CVPR 2026) [1], NeoVerse (CVPR 2026) [2], and GEN3C (CVPR 2025) [3]. Vista4D and NeoVerse address the intermediate 4D reconstructor problem by directly training the video NVS model to be robust to point cloud/4DGS artifacts (which appears to be the primary problem that explicit-prior baselines suffer from in the provided supplemental html), and GEN3C is an important paper/baseline for video NVS. The authors should compare GRAFT to these three baselines both via quantitative numbers and also qualitative results to show GRAFT's camera control and video fidelity compared to these baselines. It is difficult to evaluate GRAFT's effectiveness when it is not compared to current SOTA baselines.
  2. Visual examples with relatively 'easy' cameras: Many of the qualitative examples (in both the main paper and the supplemental) have target cameras that do not deviate significantly from the source camera. For example, in the supplemental, the first video has a target camera with a small rotation and not much parallax compared to the source video. This is in contrast to both existing baselines and missing baselines that showcase significantly more diverse and challenging cameras: GEN3C [3] shows extensive fly-throughs of large scenes, Vista4D [1] released eval datasets with significantly more dynamic videos and challenging target cameras, EX-4D reports video fidelity metrics of large viewpoint changes, etc. These are the examples which makes clear that video NVS should not be framed as merely inpainting of some rendered point cloud (which is an observation that Vista4D [1], NeoVerse [2], and DepthDirector all make) and can show the potential of GRAFT's robustness with it not needing a point cloud render. Without these more challenging examples (and comparing them to baselines), it makes it more difficult to justify the effectiveness of GRAFT.
    • Moreover, none of the visual examples seem to contain any intrinsics changes, i.e., camera zooms. Intrinsics are inherently part of the camera conditioning (i.e., point cloud/4DGS render) for explicit-prior methods so many of the baselines do support both intrinsics and extrinsics control. UCPE (GRAFT's way of camera conditioning) does encode intrinsics, so I'm wondering how GRAFT performs when there are intrinsics changes (e.g., static zoom, dolly zoom, general extrinsics + intrinsics change combinations).

[1] Kuan Heng Lin, Zhizheng Liu, Pablo Salamanca, Yash Kant, Ryan Burgert, Yuancheng Xu, Koichi Namekata, Yiwei Zhao, Bolei Zhou, Micah Goldblum, Paul Debevec, Ning Yu. "Vista4D: Video Reshooting with 4D Point Clouds." CVPR 2026.

[2] Yuxue Yang, Lue Fan, Ziqi Shi, Junran Peng, Feng Wang, Zhaoxiang Zhang. "NeoVerse: Enhancing 4D World Model with in-the-wild Monocular Videos." CVPR 2026.

[3] Xuanchi Ren, Tianchang Shen, Jiahui Huang, Huan Ling, Yifan Lu, Merlin Nimier-David, Thomas Müller, Alexander Keller, Sanja Fidler, Jun Gao. "GEN3C: 3D-Informed World-Consistent Video Generation with Precise Camera Control." CVPR 2025.

Questions for authors

Clarifications/suggestions:

  1. Terminology: The terms "appearance-first training" and "geometry-first retaking" could be seen as prioritizing one 'modality' over the other, which is not what they actually mean as implied by the paper (the -first just means which one is run/trained first). Was anything beyond ordering intended by the specific ways the authors named these two terms?
    • It'll also help to state explicitly early on that AD is an auxiliary loss used only during geometry-expert training. At first glance it can look like AD is the only loss/way that the geometry expert was trained.
  2. Eval target trajectories: How were the target trajectories for the datasets which don't have target trajectories (e.g., SpatialVID, HumanVid, DAVIS) constructed? For example: What types of trajectories were included? What magnitudes for rotation and translation were sampled? How was translation scaled relative to the scene scale/depth? Were they all randomly generated or were they hand-designed (or maybe a combination or both)?
  3. Choice of AD hyperparameters: The paper does AD on blocks {17,23,27}\{17, 23, 27\} with β=0.05\beta = 0.05 and being done every 5 optimizer steps. How were these choices made? Was there an ablation (quantitative or qualitative), or were they mostly through heuristics and quick eyeballing?
  4. Probing at other noise levels for AD: Were there any experiments probing at other noise levels for AD beyond the rationale given in the paper? Would also be nice to see perhaps an explanation or visual of how the AD attention maps start to drift with σ<1\sigma < 1.

Suggestions for future work: These are also interesting experiments that might raise my score if the authors have the bandwidth to run them (and run them)!

  1. AD applied to implicit- and explicit-prior methods: I wonder if AD can be applied to non-two-stage methods where the source video tokens and target tokens are concatenated in the sequence dimension, such as ReCamMaster for an implicit-prior method and Vista4D for an explicit-prior method. Namely, it would be cool to show AD improving performance on those methods on a matched computed budget. Though Table 3(c) shows a single-stage inference model, seeing how AD can benefit a wider variety of architectures for video NVS would be interesting.
8 comments
AuthorsOpenReview ↗

Response to Reviewer p7V7 — Part 1/3: Baseline Comparisons and Camera Control

Thank you for the detailed and constructive suggestions. They encouraged us to broaden the baseline comparison, evaluate more challenging camera settings, and test AD beyond GRAFT. We appreciate this concrete guidance and the opportunity to strengthen both the experiments and their presentation. Below, we address the main concerns and provide the corresponding results and manuscript references.

I. Vista4D, NeoVerse, and GEN3C comparisons

We added all three methods to the SpatialVID and HumanVid quantitative comparisons and to the qualitative comparisons. We also expanded Related Work to describe how Vista4D and NeoVerse train for robustness to reconstruction artifacts.

The added camera-following and fidelity results are:

DatasetMethodRot. ↓Trans. ↓FID ↓FVD ↓
SpatialVIDGEN3C1.44740.016021.14475.27
SpatialVIDNeoVerse1.61190.022544.15557.43
SpatialVIDVista4D1.29070.013622.41515.22
SpatialVIDGRAFT0.76270.010718.03439.42
HumanVidGEN3C2.7260.057740.03365.13
HumanVidNeoVerse3.3560.064050.30417.72
HumanVidVista4D2.2540.045534.79370.77
HumanVidGRAFT1.7090.040026.93346.15

SpatialVID FVD here uses I3D features. GRAFT obtains lower rotation and translation errors, FID, and FVD than these three methods on both benchmarks. The full tables also report geometric consistency, subject motion, and identity metrics.

We additionally compare against synchronized target RGB on the official iPhone benchmark and on our held-out UE set. GRAFT obtains the best average rank of 1.385 across the 13 GT-referenced scores, with the full metric-by-metric results in the revised main paper. Qualitative comparisons include human-centric videos, matched GT views in UE, and additional source videos.

Expanded in the revised manuscript: Related Work; Section 4.2 / Tables 1–3; Figure 6; and Appendix A.2.1 / Figures 9–10.

II. Challenging camera trajectories and changing intrinsics

We expanded the camera evaluation in three ways:

  • Complex camera paths: new qualitative examples show retaking under more challenging trajectories, with source and generated frames aligned in time.
  • Changing intrinsics: new examples show time-varying camera intrinsics, with the intrinsic-parameter plots displayed beneath the source and generated frames.
  • Held-out UE evaluation: 150 video pairs from 150 distinct scene regions, each containing 81 synchronized frames, include multi-person motion, crossing walks, frame exits, and coupled camera changes.

The UE trajectory mixture includes 45 dolly/tracking/lateral-translation pairs, 30 S-curve/figure-eight/crane/overhead pairs, 30 large-angle turn-away or sky-reveal return pairs, 15 coupled intrinsic–extrinsic pairs, and 30 additional orbit/reveal/tracking pairs. Average valid-warp coverage is 45.75% overall and 18.13% in the large-angle turn-away-and-return group, providing cases with substantial changes in visibility.

The updated supplementary demo includes 8 complex-trajectory examples, 4 varying-intrinsics examples, and 4 joint intrinsic–extrinsic examples, all with synchronized camera or parameter visualizations. It also presents matched baseline comparisons with Vista4D, NeoVerse, and GEN3C, and will be provided with the supplementary material.

Expanded in the revised manuscript: Appendix A.2.2 / Figure 11 for complex trajectories; Appendix A.2.3 / Figure 12 for changing intrinsics; and Appendix B.4 / Tables 12–13 and Figure 19 for the UE benchmark.

AuthorsOpenReview ↗

Response to Reviewer p7V7 — Part 2/3: AD Transfer and Training Objective

III. Extending AD to implicit- and explicit-prior methods

We appreciate this suggestion and the opportunity to test AD more broadly. Following it, we added AD to ReCamMaster and DepthDirector, covering both camera-conditioned and geometry-guided retaking models. Each integration uses the method's existing training code, with AD added as an auxiliary loss.

On SpatialVID:

MethodRot. ↓Trans. ↓Dyn-MEt3R ↑MEt3R ↓
ReCamMaster3.17810.03160.81930.2312
ReCamMaster +AD2.62000.01990.84020.1852
DepthDirector1.37340.02250.83740.1852
DepthDirector +AD1.53950.01670.86150.1792

On HumanVid:

MethodRot. ↓Trans. ↓Traj. ↑Amp. ↑F-IFS ↑
ReCamMaster5.0150.05420.2370.6140.9298
ReCamMaster +AD4.0250.06430.3450.6570.9317
DepthDirector8.1120.07700.2770.6150.9078
DepthDirector +AD4.8110.07100.3280.6400.9195

AD improves eight of the nine reported metrics for each baseline. Both models improve in SpatialVID geometric consistency and HumanVid trajectory, amplitude, and identity fidelity. For example, ReCamMaster's trajectory fidelity increases from 0.237 to 0.345, and DepthDirector's HumanVid rotation error decreases from 8.112 to 4.811.

We further compared RGB + AD with RGB + CAMEO to examine the source of the addressing gains. RGB + AD's translation-error reduction is larger in the lowest-coverage third than in the highest: 41.4% versus 7.6% on HumanVid, and 33.1% versus 15.9% on SpatialVID. SpatialVID geometric gains are also larger at low coverage. The percentages compare group-mean errors, with per-trajectory coverage defined as the median valid-warp fraction over 81 frames. These findings support stronger cross-frame source addressing when the synchronized source view provides limited direct support.

Given the rebuttal timeline, these are straightforward integrations without method-specific hyperparameter tuning. We believe the gains make AD a promising training objective for a broader range of video-retaking architectures and motivate further method-specific optimization.

Expanded in the revised manuscript: Section 4.3; Appendix A.8 / Table 9 for the AD integrations; and Appendix A.5 / Figure 16 for the coverage-dependent RGB + AD / RGB + CAMEO comparison.

IV. Terminology and AD as an auxiliary objective

We have made the ordering explicit at the first introduction of the terms:

  • Appearance-First Training: train the appearance expert first, then freeze it to supervise geometry training.
  • Geometry-First Retaking: predict target depth first, then use it for RGB synthesis.
  • AD: an auxiliary addressing loss used during geometry training, alongside the depth flow-matching loss, with objective Lgeo+βLaddr\mathcal{L}_{\mathrm{geo}}+\beta\mathcal{L}_{\mathrm{addr}}.

Expanded in the revised manuscript: Introduction and Sections 3.3–3.4; Appendix B.6 gives the update schedule and reduction details.

AuthorsOpenReview ↗

Response to Reviewer p7V7 — Part 3/3: Trajectories, Hyperparameters, and Noise Levels

V. Construction and scale normalization of target trajectories

Thank you for asking us to make this protocol explicit. SpatialVID, HumanVid, and DAVIS use the standard trajectories released by ReCamMaster, with the same Pi3X-based scale construction and target cameras supplied to all compared methods.

  • Scene radius: from the full 81-frame Pi3X depth sequence, we collect finite positive depths in a central window of at most 200 × 200 pixels per frame and set r=1.1min⁡Dr=1.1\min D. A frame with no valid central depth uses its full-image valid depths.
  • Translation amplitude: for a standard trajectory expressed relative to its first frame, let LL be its maximum translation magnitude. Translating paths are scaled by α=0.3r/L\alpha=0.3r/L, giving a maximum displacement of 0.3r. Orientations look toward the scene origin. Zero-translation templates use their rotation curves to generate constant-radius orbits.
  • Common coordinates: source and target cameras undergo the same rigid transformation so that the first source camera is the identity. Depth and camera translation retain their shared Pi3X reconstruction units within each scene; intrinsics follow the actual resize and crop.
  • Per-model reconstruction: when a method reconstructs its own source geometry, scale alignment maps the fixed target cameras into its reconstruction coordinates, preserving the benchmark trajectory definition.

The held-out UE benchmark is constructed separately in the engine:

  1. We sample navigable scene regions and create multi-person events with fixed actor identities, animations, and event timing.
  2. Camera paths are planned around actor/region anchors; following paths update their anchors with the actors' actual motion. Candidate seeds, path distances, amplitudes, and curvatures are recorded.
  3. We check camera/environment and camera/actor collisions using continuous sweeps and a 25 cm camera sphere, together with actor support, collisions, speed, and the intended visibility changes for return trajectories.
  4. Paired videos replay the same event using frozen per-frame camera positions, rotations, and intrinsics. UE150 selects one pair from each of 150 independent regions, retaining 81 frames, the same initial source/target pose and intrinsics, and continuous landscape motion.

UE camera positions are converted from centimeters to meters, and the camera axes are transformed to the model convention, consistently with metric GT depth. The paths retain their engine-planned motion, while the 0.3r rule applies to the three reconstructed monocular benchmarks.

Expanded in the revised manuscript: Section 4.1; Appendix B.2 for scene scale and standard trajectories; Appendix B.4 / Tables 12–13 and Figure 19 for held-out UE construction.

VI. Selection of AD hyperparameters

We used a simple heuristic strategy to select the AD settings, guided by attention diagnostics on training examples across blocks 15–29 at σ = 1. We chose three blocks with complementary properties: block 17 has the strongest temporal correlation (0.897), block 23 assigns the largest attention mass to source tokens (31.51%), and block 27 has the strongest same-time argmax rate (92.86%). Their losses receive equal weights. We set β = 0.05 based on the initial magnitudes of the KL and flow-matching losses, and apply AD every five optimizer steps to balance supervision frequency and training cost. The reported results use these heuristic settings without extensive hyperparameter tuning; more systematic tuning of the block selection, loss weight, and update frequency may yield further improvements.

Expanded in the revised manuscript: Appendix B.8 / Table 15 for the block-selection analysis and Appendix B.6 for the exact loss and update schedule.

VII. Probing at other noise levels

We added RNT, which evaluates the addressing KL loss at randomly sampled noise levels, while the teacher and student share the sampled level and Gaussian noise. GRAFT uses the pure-noise endpoint.

DatasetKL probingRot. ↓Trans. ↓
SpatialVIDRandom noise level (RNT)1.2800.015
SpatialVIDPure-noise endpoint (GRAFT)0.7630.011
HumanVidRandom noise level (RNT)3.1250.054
HumanVidPure-noise endpoint (GRAFT)1.7090.040

On HumanVid, pure-noise probing also increases trajectory fidelity from 0.361 to 0.379 and amplitude fidelity from 0.655 to 0.676. These results support the motivation for the pure-noise endpoint: it removes residual target RGB from the teacher's noisy input, allowing its source addressing to be learned under the source observations and known target layout. RNT is presented as a probing-noise diagnostic and is excluded from the ablation rankings and best/second-best markings.

Expanded in the revised manuscript: Section 4.3 / Table 4, variants (f) RNT and (g) GRAFT; Section 3.2 explains the probing rationale.

Reviewer_tP65OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The core of this paper is "Addressing Distillation." First, it trains an "appearance expert" to take a source video and ground-truth target depth as input and output an RGB image from the target viewpoint. Once trained, this expert is frozen, and its attention distribution is analyzed at time steps involving pure noise. This attention distribution is then distilled, via KL divergence, into a "geometry expert" tasked with predicting the target depth. A key advantage is that no manual geometric annotations are required. During inference, the geometry expert first generates the depth, which is then used by the appearance expert to synthesize the RGB output. Across four benchmarks, the method consistently outperforms six baselines in terms of camera-motion following, geometric consistency, motion preservation, and identity retention.

Critical strengths
  1. The experiments are comprehensive. Metrics cover camera-motion following, geometric consistency, fidelity, motion preservation, facial identity, and VBench. Ablation studies address the vast majority of the model's components.
  2. Using the model’s own internal attribution (attention) as a supervisory signal eliminates the need for extensive geometric annotations.
  3. The writing is clear, the notation is standardized, and the logical flow is sound.
Critical weaknesses
  1. Some data points require further verification; for instance, in Table 6 of the appendix, the rotation error for "DepthDirector" is unusually high.
  2. As the authors themselves acknowledge, the model's performance in real-world domains remains unclear. Given that the paper’s central premise is "avoiding ground-truth depth supervision," testing the method on real-world data, where reliable geometric supervision is truly unavailable, would provide a more compelling validation of its effectiveness.
  3. The "geometry expert predicts depth, then appearance expert synthesizes RGB" pipeline is a common approach, but it is often prone to cumulative error. Does GRAFT exhibit similar issues? Would adopting a framework like GaC [1] yield better results?

[1] Geometry-as-context: Modulating Explicit 3D in Scene-consistent Video Generation to Geometry Context. CVPR 2026.

Questions for authors

Beyond the points raised in the "Weaknesses" section, it is worth noting that during training, the depth for the first frame is derived from UE5, whereas during inference, it is estimated using MoGe+Pi3X. This discrepancy may introduce a "train-test gap". I suggest the authors analyze this.

6 comments
AuthorsOpenReview ↗

Response to Reviewer tP65 — Part 1/2: DAVIS and Real-World Evaluation

Thank you for the thoughtful review and for drawing our attention to the DAVIS result, real-world validation, and the depth-to-RGB dependency. Your comments helped us make the evaluation more informative and the training and inference protocol clearer. We have added targeted analyses for these points and summarize the findings below.

I. DepthDirector's rotation error on DAVIS

Thank you for flagging this result. We rechecked the preprocessing, camera-coordinate transformations, source-video assignments, and source/depth/mask conditioning used for DepthDirector. These evaluations use the same core geometry-processing and inference pipeline across DAVIS, HumanVid, and SpatialVID. We then analyzed the input warp conditions for the 300 DAVIS and 300 HumanVid cases.

The new analysis shows:

  • Lower geometric support on DAVIS: median warp coverage is 51.48%, compared with 60.35% on HumanVid. Coverage is the per-video median valid-mask fraction over 81 frames.
  • More extreme cases: 42 DAVIS samples have coverage below 10%, compared with one HumanVid sample.
  • Larger errors in the extreme subset: the 42 low-coverage DAVIS cases have a mean rotation error of 43.56°, compared with 21.11° over all DAVIS cases.
  • Consistent trends: lower warp coverage is associated with larger camera errors and lower semantic and subject consistency in both datasets. The plot reports coverage-bin means and scene-level bootstrap confidence intervals.

Visual inspection of these low-coverage cases shows sparse or unreliable warped content and strongly degraded outputs. The analysis connects the high DAVIS error to a concentration of difficult input conditions with weak geometric support.

Expanded in the revised manuscript: Appendix A.3.1 / Figure 13, accompanying the original DAVIS comparison in Table 7.

II. Real-world evaluation and the role of GT depth in training

We appreciate the emphasis on real-world validation. We have added evaluation on the full official iPhone benchmark, using synchronized target-camera RGB. The revised evaluation includes full-frame and masked reconstruction metrics, as well as FID and FVD against GT target videos. The masked metrics use the non-hole regions of projected point clouds.

MethodPSNR / mPSNR ↑SSIM / mSSIM ↑LPIPS / mLPIPS ↓FID ↓FVD ↓
ReCamMaster10.58 / 10.630.266 / 0.6490.810 / 0.633132.211937.34
Vista4D13.57 / 13.540.299 / 0.6680.578 / 0.45683.55924.21
GRAFT13.16 / 13.570.300 / 0.6730.616 / 0.464105.63843.80

GRAFT achieves higher masked PSNR and SSIM and lower FVD than these baselines. We also added a held-out UE benchmark with 150 source–target pairs from 150 distinct scene regions to evaluate the same model under diverse actor events and camera paths.

We have clarified the training objective early in the paper: GRAFT uses rendered GT depth during training, and AD is an auxiliary loss that teaches source retrieval alongside depth-generation supervision. At inference, the first-frame reference is estimated with MoGe-3 and Pi3X, and the geometry expert predicts the target-depth sequence. The contribution is the transfer of addressing learned by the depth-conditioned appearance expert to the geometry predictor.

We also examined where this retrieval supervision helps by comparing RGB + AD with RGB + CAMEO under comparable training compute. The translation-error reduction is 41.4% versus 7.6% in the lowest- and highest-coverage thirds on HumanVid, and 33.1% versus 15.9% on SpatialVID. These are relative reductions in each group's mean error; coverage is the median valid-warp fraction over 81 frames. Larger gains when the synchronized source view supplies less direct support are consistent with the role of AD in learning to retrieve complementary information across source frames.

Expanded in the revised manuscript: Abstract, Introduction, and Section 3.3; Section 4.2 / Table 3; Appendix B.3 for the iPhone protocol and Appendix B.4 for held-out UE; and Appendix A.5 / Figure 16 for the coverage-dependent addressing analysis.

AuthorsOpenReview ↗

Response to Reviewer tP65 — Part 2/2: Depth Sensitivity, Error Propagation, and GaC

III. Error propagation, the anchor train–test difference, and GaC

We added experiments that distinguish anchor-scale sensitivity from errors in the complete target-depth condition. The source video, cameras, and appearance expert are fixed in these comparisons:

  • The anchor-scale rows multiply the inference-time estimated first-frame reference by 1, 1.5, or 2 and rerun geometry prediction.
  • Noisy Depth perturbs the standard pipeline's complete predicted target-depth condition at σ = 0.2 and 0.4 before RGB synthesis.
  • Oracle supplies MoGe-3 depth estimated from the full target video and aligned to the Pi3X scale to the same appearance expert.
ConditionPSNR / mPSNR ↑SSIM / mSSIM ↑LPIPS / mLPIPS ↓
Anchor ×113.16 / 13.570.2999 / 0.67310.6159 / 0.4635
Anchor ×1.513.37 / 13.330.2943 / 0.66670.6210 / 0.4917
Anchor ×212.98 / 13.040.2948 / 0.66720.6477 / 0.5145
Predicted depth + noise (σ = 0.2)13.05 / 13.070.2784 / 0.65830.6454 / 0.5098
Predicted depth + noise (σ = 0.4)12.57 / 12.660.2586 / 0.64890.6713 / 0.5309
Full-video reference depth (Oracle)15.67 / 16.690.3729 / 0.71360.4682 / 0.3372

The tested anchor-scale changes have a modest effect on reconstruction quality. Tests with noise on the complete predicted depth condition show PSNR of 13.05 and 12.57 and LPIPS of 0.6454 and 0.6713 at σ = 0.2 and 0.4, respectively, compared with 13.16 and 0.6159 for the unperturbed condition. Supplying the full-video reference depth improves PSNR by 2.51 dB and decreases LPIPS to 0.4682. These results make the depth-to-RGB dependency explicit and identify target-depth quality as a route to improving the output. The revision also explains how rendered training anchors and externally estimated inference anchors enter the same conditioning mechanism.

GaC uses geometry as context in an interleaved scene-generation framework. GRAFT focuses on retrieving evidence from an observed dynamic video, with geometry prediction followed by depth-conditioned appearance synthesis. We view geometry context and AD as complementary: geometry context organizes the interaction between geometry and RGB, while AD supervises which source evidence to retrieve. Combining these ideas is a promising direction for strengthening the interaction between stages, motivated by the gains from the new reference-depth experiment.

Expanded in the revised manuscript: Appendix A.9 / Table 10; Appendix B.5 for anchor construction and conditioning; and Appendix G / Figure 23 for cascade limitations and representative failures.

AuthorsOpenReview ↗

Follow-up on the DAVIS Analysis and Depth Sensitivity Experiments

Dear Reviewer tP65,

Thank you again for your careful review. We are following up with a brief summary of the analyses and clarifications added in response to your concerns:

  • DepthDirector on DAVIS: We checked preprocessing, camera transformations, and conditioning, and added a coverage analysis on 300 DAVIS and 300 HumanVid cases. DAVIS contains 42 cases with coverage below 10%, versus one on HumanVid; the analysis relates these difficult input conditions to larger errors.
  • Real-world validation and training depth: We added evaluation on the full official iPhone benchmark against synchronized target RGB. We also clarified that rendered GT depth is used during training, while standard inference uses an estimated first-frame anchor and predicted target depth.
  • Depth-to-RGB dependency: We added anchor-scale, predicted-depth-noise, and full-video reference-depth tests to quantify sensitivity and the benefit of improving target depth, together with the discussion of GaC and representative failures.

When you have a chance, we would appreciate hearing whether these additions address your concerns about the DAVIS result, real-world validation, and the depth-to-RGB dependency. Please let us know if any point needs further clarification.

Reviewer_LUpGOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes GRAFT, a video retaking framework that re-renders a dynamic source video along a new camera trajectory. The core idea is Addressing Distillation (AD).

An appearance expert is first trained to synthesize the target RGB video from the source video and ground-truth (GT) target depth. This expert is then frozen and probed at the pure-noise endpoint to extract its target-to-source attention distributions. These distributions supervise the attention of a geometry expert that predicts target-view depth. At inference, the geometry expert first generates the target depth, and the same appearance expert synthesizes the RGB video conditioned on it (Geometry-First Retaking).

The paper reports improvements in camera trajectory following, geometric consistency, and subject motion preservation on SpatialVID, HumanVid, DAVIS, and iPhonePTZ.

Critical strengths
  • The idea of learning where to retrieve source evidence from an appearance synthesis task with known geometry, and transferring it to geometry prediction, is novel. The motivation is clear: depth generation supervision alone does not specify how the predictor should use the source video.

  • The attention analysis in Fig. 10 shows that a depth-only predictor over-relies on the first source frame and fails to preserve subsequent subject motion. This connects the motivation of the method to its observed effect.

  • The implementation details of AD (attention recomputation, the order of head averaging and KL reduction, block selection, and the update schedule) are described very specifically, which supports reproducibility.

  • The evaluation is broad, covering multiple benchmarks, iPhonePTZ with GT target videos, VBench, and a user study, and the method shows consistent improvements on most metrics.

Critical weaknesses

W1. Missing comparison with correspondence-based attention supervision. The paper repeatedly emphasizes that AD learns addressing "without constructing explicit geometric correspondence labels." However, GRAFT's training data already provides GT metric depth and camera parameters, from which target-to-source correspondences can be computed directly by reprojection. In this setting, not using correspondence labels is therefore not a practical advantage in itself.

The key question is whether the teacher's soft attention is a better supervision signal than correspondences computed from GT geometry. I request an ablation that keeps the GRAFT pipeline unchanged and replaces only the target distribution of L_addr with one derived from GT depth reprojection (i.e., geometric correspondence-based attention supervision, as in CAMEO [1] and MVTrack4Gen [2]). If AD performs better, an analysis of the situations where soft distributions help (e.g., occlusion or retrieval from multiple frames) would further strengthen the paper.

[1] Kwon et al., "Cameo: Correspondence-attention alignment for multi-view diffusion models.", CVPR 2026.

[2] Lee et al., "MVTrack4Gen: Multi-View Point Tracking as Geometric Supervision for 4D Video Generation.", arXiv 2026.

W2. Missing comparison with and discussion of Vista4D. Vista4D [3], which addresses the same video reshooting task, is neither cited nor compared. Its code, models, and evaluation data are all publicly available, so a comparison is feasible.

This omission is particularly important because the paper uses the sensitivity of explicit geometry-based methods to reconstruction quality as a main motivation, and supports this with Ablation II (warp-based variants). Vista4D[3] directly addresses this issue by training on noisy 4D reconstructions. The warp-completion variants in Ablation II are therefore not sufficient to conclude that GRAFT outperforms explicit-geometry approaches in general. I request quantitative and qualitative comparisons with Vista4D [3] , along with a discussion in the related work.

[3] Lin et al., "Vista4D: Video Reshooting with 4D Point Clouds.", CVPR 2026.

W3. Insufficient GT-referenced evaluation (PSNR, SSIM, LPIPS). The main benchmarks (SpatialVID, HumanVid, DAVIS) have no GT for the target views. As a result, FID, FVD, and CLIP-V are all measured against the source video, and how well the generated results match the actual target views is not directly verified. GT-referenced metrics (PSNR, SSIM, LPIPS) are reported only on 40 iPhonePTZ pairs in the appendix, which is limited to the different-initial-viewpoint extension and is small in scale.

I request evaluation on the following data where GT-referenced evaluation is possible:

  • The iPhone dataset [4] , a real-world dynamic dataset with synchronized multi-view evaluation cameras. Since generative models synthesize regions unobserved in the source, the co-visibility masked metrics proposed in that paper (mPSNR, mSSIM, mLPIPS) should also be reported.

  • Held-out synthetic data (e.g., rendered with Unreal Engine) from scenes disjoint from the training set, to evaluate diverse camera trajectories.

[4] Gao et al., "Monocular Dynamic View Synthesis: A Reality Check.", NeurIPS 2022.

W4. No sensitivity analysis to depth errors. The appearance expert relies entirely on the predicted depth, without camera conditioning. The robustness of the final output to depth errors is therefore critical, but it is not analyzed. I request the following analyses, for example on held-out synthetic data where GT depth is available:

  • Oracle comparison: feed GT depth and predicted depth into the same appearance expert to quantify the performance gap attributable to depth prediction.

  • Scale error: the effect of global scale errors applied to GT depth. This is particularly relevant since metric depth is used.

  • Robustness to errors in the first-frame depth reference a: needed because a relies on external estimators (MoGe-3, Pi3X).

For each case, please report camera accuracy, geometric consistency, and RGB fidelity as a function of perturbation strength.

W5. Insufficient discussion of limitations. The limitations appear only briefly in Appendix G and are limited to generic issues (scenes with little geometric variation, synthetic training data). Limitations specific to the proposed method are not discussed:

  • Geometry-First Retaking is a one-directional cascade, so errors in the predicted depth propagate directly to the RGB output without any correction mechanism.
  • No failure cases are shown.

I encourage the authors to move the limitations into the main paper and include representative failure cases.

W6. Missing details on computational resources. The paper reports the number of GPUs (8) but not the GPU type, total training time, or GPU-hours. This makes the timing numbers in Appendix B.7 difficult to interpret or compare with other methods. Please specify the hardware and report the total computational cost of both training stages and inference.

Overall Assessment The idea of distilling source addressing from an appearance expert is novel, and the results are strong across benchmarks. However, it remains unclear whether AD is a better supervision signal than geometric correspondences (W1), the paper lacks a comparison with Vista4D, the most closely related explicit-geometry method (W2), and GT-referenced evaluation is limited (W3). I am willing to raise my score if the authors convincingly address W1, W2, and W3 in the rebuttal.

Questions for authors
  1. How does performance change if correspondences derived from GT depth reprojection are used as the target distribution of L_addr instead of the teacher's attention? (W1)

  2. How does GRAFT compare with Vista4D? (W2)

  3. What are the results in mPSNR, mSSIM, and mLPIPS on the iPhone dataset or on held-out synthetic data? (W3)

  4. What is the performance gap relative to an oracle that feeds GT depth into the appearance expert? How sensitive are the results to depth scale errors and to errors in the depth reference a? (W4)

  5. What GPU type was used, and what is the total computational cost of training and inference? (W6)

7 comments
AuthorsOpenReview ↗

Response to Reviewer LUpG — Part 1/3: Addressing Supervision and Coverage Analysis

Thank you for the careful and constructive review. Your questions about correspondence supervision, Vista4D, and GT-referenced evaluation helped us strengthen the experimental comparison and clarify the role of learned addressing. We appreciate the opportunity to improve the work through these suggestions. Below, we summarize the added experiments and results, followed by their locations in the revised manuscript.

I. Correspondence supervision and coverage-dependent gains (W1)

Thank you for raising this important comparison. Following your suggestion, we implemented CAMEO's correspondence-attention supervision in our codebase and trained CAMEO* with the same pretrained Wan2.2 TI2V-5B backbone and UE training data as GRAFT, since an official pretrained checkpoint was unavailable for our evaluation.

Appendix B.7 provides the implementation details, including correspondence construction, attention supervision, and the training configuration. Figure 20 shows CAMEO*'s training-loss curve as a function of training time.

The comparison includes:

  • RGB + CAMEO (CAMEO*): attention supervision from rendered depth and calibrated cameras, using nearest-neighbor geometric matches and cycle-consistency filtering.
  • RGB + AD: the direct RGB-generation variant from our ablation, with learned addressing supervision. Its training compute is comparable to CAMEO*, making it a useful comparison of the supervision signals.
  • GRAFT: the full geometry-first model, included as a reference for the complete framework.

We focus on camera following and geometric consistency on SpatialVID:

MethodRot. ↓Trans. ↓Dyn-MEt3R ↑MEt3R ↓
RGB + CAMEO (CAMEO*)1.47970.01750.86260.1823
RGB + AD1.1740.0120.8780.176
GRAFT0.76270.01070.87770.1745

On HumanVid, we report camera following and subject-motion preservation:

MethodRot. ↓Trans. ↓Traj. ↑Amp. ↑
RGB + CAMEO (CAMEO*)3.6980.06880.3350.637
RGB + AD2.1160.0490.3790.678
GRAFT1.7090.04000.3790.676

RGB + AD improves all eight reported metrics relative to CAMEO under comparable training compute.* GRAFT further reduces the rotation and translation errors on both datasets.

Following your suggestion to examine occlusion and retrieval across frames, we additionally compared RGB + AD and RGB + CAMEO across warp-coverage levels. Low coverage means that the source frame at the target time directly supports less of the target view, increasing the need for complementary evidence from other source frames.

The new analysis shows larger gains under limited coverage:

DatasetTranslation-error reduction: lowest-coverage thirdHighest-coverage third
HumanVid41.4%7.6%
SpatialVID33.1%15.9%

These percentages compare the two methods' mean errors within each coverage group. Each trajectory's coverage is the median valid-warp fraction over its 81 frames. On HumanVid, coverage is negatively correlated with the paired absolute translation-error improvement (Spearman ρ = −0.428, 95% CI [−0.634, −0.180]).

SpatialVID geometry shows the same low-coverage advantage:

MetricAbsolute gain: lowest-coverage thirdHighest-coverage third
Dyn-MEt3R increase0.02310.0127
MEt3R decrease0.00920.0009

The figure summarizes each source clip across its five target trajectories; the additional geometry analysis uses individual trajectories. Confidence intervals bootstrap original source videos, retaining their clips and trajectories together.

CAMEO supervises synchronized geometric matches, while AD transfers an addressing distribution over source regions and times. The larger translation and geometric gains where synchronized support is sparse support more effective cross-frame source addressing with AD. The attention visualization further illustrates reduced initial-frame bias and preservation of subsequent subject motion.

Expanded in the revised manuscript: Section 4.3 for the brief summary; Appendix A.5 / Figure 16 for the coverage-dependent comparison; Appendix A.6 / Figure 17 for the source-addressing visualization; and Appendix B.7 / Table 14 / Figure 20 for the CAMEO implementation, aggregate comparison, and training-loss curve. Tables 1–2 report the full benchmark results.

AuthorsOpenReview ↗

Response to Reviewer LUpG — Part 2/3: Vista4D and Ground-Truth Evaluation

II. Vista4D comparison and explicit geometry guidance (W2)

Thank you for drawing our attention to Vista4D. We added it to the quantitative and qualitative comparisons and expanded the related-work discussion to describe its use of reconstructed multiview training data for handling point-cloud artifacts. The new evaluation also includes NeoVerse and GEN3C.

Key results for Vista4D and GRAFT are:

DatasetMethodRot. ↓Trans. ↓FID ↓FVD ↓
SpatialVIDVista4D1.29070.013622.41515.22
SpatialVIDGRAFT0.76270.010718.03439.42
HumanVidVista4D2.2540.045534.79370.77
HumanVidGRAFT1.7090.040026.93346.15

GRAFT has lower camera errors, FID, and FVD on both benchmarks. We additionally compare the methods against synchronized target RGB on iPhone and held-out UE, as detailed below. On iPhone, the methods have complementary strengths: Vista4D achieves lower LPIPS and FID, while GRAFT achieves higher masked PSNR and SSIM and lower FVD.

The quantitative comparisons are accompanied by matched qualitative examples on human-centric videos, held-out UE scenes, and additional real videos. The updated supplementary demo includes matched baseline comparisons, complex trajectories, varying intrinsics, and joint intrinsic–extrinsic changes, with synchronized camera visualizations. It will be provided with the supplementary material.

Expanded in the revised manuscript: Related Work and Section 4.2; Tables 1–3; Figure 6; and Appendix A.2.1 / Figures 9–10. The warp-completion ablation is discussed as a comparison between the specific conditioning variants in Table 4.

III. GT-referenced evaluation on iPhone and held-out UE (W3)

Your suggestion prompted us to expand the target-view evaluation in two settings:

  • Real-world iPhone: the full official benchmark, using synchronized target-camera RGB. We report full-frame PSNR/SSIM/LPIPS and masked mPSNR/mSSIM/mLPIPS; the masks are the non-hole regions of projected point clouds.
  • Held-out UE: 150 source–target video pairs from 150 distinct scene regions, disjoint from training, with 81 synchronized frames per video. The set includes three- and five-person scenes, crossing motions, frame exits, complex camera paths, and coupled intrinsic–extrinsic changes. We sample navigable regions, replay synchronized actor events under programmatically planned and collision-checked paths, and render from frozen per-frame cameras.
  • Target-referenced distribution metrics: FID and FVD are computed against GT target videos in both settings.

Representative iPhone results are:

MethodmPSNR ↑mSSIM ↑mLPIPS ↓FID ↓FVD ↓
ReCamMaster10.630.6490.633132.211937.34
GEN3C10.250.6230.638176.181950.58
Vista4D13.540.6680.45683.55924.21
GRAFT13.570.6730.464105.63843.80

The corresponding held-out UE results are:

MethodPSNR ↑SSIM ↑LPIPS ↓FID ↓FVD ↓
ReCamMaster15.800.4800.45041.47236.49
GEN3C17.670.5510.34341.26191.72
Vista4D17.720.5230.31228.17138.47
GRAFT18.680.5390.28425.93126.63

GRAFT obtains the best average rank of 1.385 across the 13 reported scores, compared with 1.769 for Vista4D. The UE set also evaluates substantially changing visibility: average valid-warp coverage is 45.75%, falling to 18.13% for large-angle turn-away-and-return trajectories.

Expanded in the revised manuscript: Sections 4.1–4.2 and Table 3; Appendix B.3 for the evaluation protocol; Appendix B.4 / Tables 12–13 for UE data and trajectory composition; and Figure 19 for paired-rendering examples.

AuthorsOpenReview ↗

Response to Reviewer LUpG — Part 3/3: Depth Diagnostics, Failure Cases, and Compute

IV. Depth prediction, anchor scale, and depth-condition errors (W4)

We added three diagnostics on iPhone, keeping the source clips, cameras, and appearance expert fixed:

  • First-frame anchor scale: multiply the estimated anchor by 1, 1.5, or 2, then rerun the full geometry-to-RGB pipeline.
  • Corrupted predicted depth: perturb the complete target-depth condition predicted by the standard pipeline at noise levels σ = 0.2 and 0.4 before supplying it to the appearance expert.
  • Full-video reference depth (Oracle): replace predicted target depth with MoGe-3 depth estimated from the full target video and aligned to the Pi3X scale.
ConditionPSNR / mPSNR ↑SSIM / mSSIM ↑LPIPS / mLPIPS ↓
Anchor ×113.16 / 13.570.2999 / 0.67310.6159 / 0.4635
Anchor ×1.513.37 / 13.330.2943 / 0.66670.6210 / 0.4917
Anchor ×212.98 / 13.040.2948 / 0.66720.6477 / 0.5145
Predicted depth + noise (σ = 0.2)13.05 / 13.070.2784 / 0.65830.6454 / 0.5098
Predicted depth + noise (σ = 0.4)12.57 / 12.660.2586 / 0.64890.6713 / 0.5309
Full-video reference depth (Oracle)15.67 / 16.690.3729 / 0.71360.4682 / 0.3372

Anchor scaling has a modest effect over the tested multipliers, with full-frame PSNR ranging from 12.98 to 13.37. The depth-noise tests show a gradual reduction in reconstruction quality: PSNR changes from 13.16 to 13.05 and 12.57 at σ = 0, 0.2, and 0.4, respectively, while LPIPS changes from 0.6159 to 0.6454 and 0.6713. The full-video reference-depth condition raises PSNR from 13.16 to 15.67 and lowers LPIPS from 0.6159 to 0.4682, identifying a concrete opportunity to improve the final video through better target-depth prediction.

Expanded in the revised manuscript: the depth-sensitivity discussion in Section 4.2; Appendix A.9 / Table 10 for the experiments; and Appendix B.5 for anchor construction and conditioning.

V. Cascade limitations and representative failures (W5)

We now discuss the depth-to-RGB dependency explicitly: errors in predicted target depth can propagate to RGB synthesis, and the present cascade has no RGB-to-depth feedback. The depth-condition diagnostics above quantify the effect of changing the intermediate representation.

We also added three representative failures with source RGB, predicted depth, and retaken RGB:

  • Abstract motion graphics: predicted depth has little discernible structure, and the retaken video loses the source dynamics.
  • Screen-space overlays: watermark or broadcast-logo shapes become distorted during retaking.
  • Translucent and emissive effects: scene depth captures the people and background, while the effect's fine structure and temporal evolution are incompletely preserved.

Expanded in the revised manuscript: Appendix G, “Limitations and Future Work,” and Appendix G.1 / Figure 23, together with Appendix A.9 / Table 10.

VI. Computational resources (W6)

Both training stages use eight NVIDIA A100 GPUs:

StageStepsTimeGPU-hours
Appearance20,00036.67 h293.33
Geometry, including AD30,00068.67 h549.33
Total50,000105.33 h842.67

Appearance steps take 6.6 s. Geometry steps take 6.8 s ordinarily and 14 s on each fifth step with KL distillation, giving a mean of 8.24 s per step. Inference uses one A100, 30 denoising steps per expert, 92 s end-to-end latency, and 15 GB peak GPU memory for an 81-frame, 384 × 704 video.

Expanded in the revised manuscript: Appendix B.12 / Table 16.

Comments on the paper

1 comment

Dear Reviewers,

As the discussion phase comes to an end, we'd appreciate it if you could take a moment to review the authors' responses. If you have any remaining concerns or questions, please raise them with the authors before the discussion closes.

Best,

Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.