- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
GRAFT breaks down video novel-view synthesis (NVS) (or video retaking) into two stages: Target depth synthesis (based on the source video and target camera trajectory) and appearance transfer (transferring the appearance of the source video onto some given depth video). During training, the appearance expert is first trained with source-target video pairs (along with the target depth). The geometry expert is then trained with addressing distillation, which guides it to mirror the source-target attention map of the trained and frozen appearance expert. During inference, the geometry expert is first run to obtain target depths before using the appearance expert to transfer the source video's appearance onto said target depth. Doing so removes the need to do per-frame 4D point cloud renders that can introduce artifacts from 4D reconstruction which explicit-prior methods can suffer from (e.g., TrajectoryCrafter), while improving camera control precision compared to implicit-prior methods (e.g., ReCamMaster).
Critical strengths
- Hints of generality of addressing distillation (AD): The authors show in the ablations that AD not only helps the two-stage framework of GRAFT but also helps when the camera condition is warped target-view depth (which on the surface seems to require less source-target correspondence as the model is partially 'inpainting' that target-view depth). This has the potential wider implication that training for source-target correspondence is just generally a good 'representation learning' objective that can be used to distill/supervise video NVS models which require this correspondence more explicitly.
- Two-stage pipeline being less explicit: The two-stage pipeline of GRAFT makes its approach potentially more scalable as it is not constrained by the quality of the point cloud render (from the 4D reconstructor) which many baselines do. Moreover, compared to implicit-prior methods (e.g., ReCamMaster, CamCloneMaster), explicit depth grounding with the depth anchor resolves the scale ambiguity of camera extrinsics/reference videos.
Critical weaknesses
- Missing baselines: The paper is missing several important baselines which have been shown to outperform existing baselines in the paper (ReCamMaster, TrajectoryCrafter, CamCloneMaster, and EX-4D). Namely, the paper is missing comparisons to Vista4D (CVPR 2026) [1], NeoVerse (CVPR 2026) [2], and GEN3C (CVPR 2025) [3]. Vista4D and NeoVerse address the intermediate 4D reconstructor problem by directly training the video NVS model to be robust to point cloud/4DGS artifacts (which appears to be the primary problem that explicit-prior baselines suffer from in the provided supplemental html), and GEN3C is an important paper/baseline for video NVS. The authors should compare GRAFT to these three baselines both via quantitative numbers and also qualitative results to show GRAFT's camera control and video fidelity compared to these baselines. It is difficult to evaluate GRAFT's effectiveness when it is not compared to current SOTA baselines.
- Visual examples with relatively 'easy' cameras: Many of the qualitative examples (in both the main paper and the supplemental) have target cameras that do not deviate significantly from the source camera. For example, in the supplemental, the first video has a target camera with a small rotation and not much parallax compared to the source video. This is in contrast to both existing baselines and missing baselines that showcase significantly more diverse and challenging cameras: GEN3C [3] shows extensive fly-throughs of large scenes, Vista4D [1] released eval datasets with significantly more dynamic videos and challenging target cameras, EX-4D reports video fidelity metrics of large viewpoint changes, etc. These are the examples which makes clear that video NVS should not be framed as merely inpainting of some rendered point cloud (which is an observation that Vista4D [1], NeoVerse [2], and DepthDirector all make) and can show the potential of GRAFT's robustness with it not needing a point cloud render. Without these more challenging examples (and comparing them to baselines), it makes it more difficult to justify the effectiveness of GRAFT.
- Moreover, none of the visual examples seem to contain any intrinsics changes, i.e., camera zooms. Intrinsics are inherently part of the camera conditioning (i.e., point cloud/4DGS render) for explicit-prior methods so many of the baselines do support both intrinsics and extrinsics control. UCPE (GRAFT's way of camera conditioning) does encode intrinsics, so I'm wondering how GRAFT performs when there are intrinsics changes (e.g., static zoom, dolly zoom, general extrinsics + intrinsics change combinations).
[1] Kuan Heng Lin, Zhizheng Liu, Pablo Salamanca, Yash Kant, Ryan Burgert, Yuancheng Xu, Koichi Namekata, Yiwei Zhao, Bolei Zhou, Micah Goldblum, Paul Debevec, Ning Yu. "Vista4D: Video Reshooting with 4D Point Clouds." CVPR 2026.
[2] Yuxue Yang, Lue Fan, Ziqi Shi, Junran Peng, Feng Wang, Zhaoxiang Zhang. "NeoVerse: Enhancing 4D World Model with in-the-wild Monocular Videos." CVPR 2026.
[3] Xuanchi Ren, Tianchang Shen, Jiahui Huang, Huan Ling, Yifan Lu, Merlin Nimier-David, Thomas Müller, Alexander Keller, Sanja Fidler, Jun Gao. "GEN3C: 3D-Informed World-Consistent Video Generation with Precise Camera Control." CVPR 2025.
Questions for authors
Clarifications/suggestions:
- Terminology: The terms "appearance-first training" and "geometry-first retaking" could be seen as prioritizing one 'modality' over the other, which is not what they actually mean as implied by the paper (the -first just means which one is run/trained first). Was anything beyond ordering intended by the specific ways the authors named these two terms?
- It'll also help to state explicitly early on that AD is an auxiliary loss used only during geometry-expert training. At first glance it can look like AD is the only loss/way that the geometry expert was trained.
- Eval target trajectories: How were the target trajectories for the datasets which don't have target trajectories (e.g., SpatialVID, HumanVid, DAVIS) constructed? For example: What types of trajectories were included? What magnitudes for rotation and translation were sampled? How was translation scaled relative to the scene scale/depth? Were they all randomly generated or were they hand-designed (or maybe a combination or both)?
- Choice of AD hyperparameters: The paper does AD on blocks with and being done every 5 optimizer steps. How were these choices made? Was there an ablation (quantitative or qualitative), or were they mostly through heuristics and quick eyeballing?
- Probing at other noise levels for AD: Were there any experiments probing at other noise levels for AD beyond the rationale given in the paper? Would also be nice to see perhaps an explanation or visual of how the AD attention maps start to drift with .
Suggestions for future work: These are also interesting experiments that might raise my score if the authors have the bandwidth to run them (and run them)!
- AD applied to implicit- and explicit-prior methods: I wonder if AD can be applied to non-two-stage methods where the source video tokens and target tokens are concatenated in the sequence dimension, such as ReCamMaster for an implicit-prior method and Vista4D for an explicit-prior method. Namely, it would be cool to show AD improving performance on those methods on a matched computed budget. Though Table 3(c) shows a single-stage inference model, seeing how AD can benefit a wider variety of architectures for video NVS would be interesting.
8 comments
Response to Reviewer p7V7 — Part 2/3: AD Transfer and Training Objective
III. Extending AD to implicit- and explicit-prior methods
We appreciate this suggestion and the opportunity to test AD more broadly. Following it, we added AD to ReCamMaster and DepthDirector, covering both camera-conditioned and geometry-guided retaking models. Each integration uses the method's existing training code, with AD added as an auxiliary loss.
On SpatialVID:
| Method | Rot. ↓ | Trans. ↓ | Dyn-MEt3R ↑ | MEt3R ↓ |
|---|---|---|---|---|
| ReCamMaster | 3.1781 | 0.0316 | 0.8193 | 0.2312 |
| ReCamMaster +AD | 2.6200 | 0.0199 | 0.8402 | 0.1852 |
| DepthDirector | 1.3734 | 0.0225 | 0.8374 | 0.1852 |
| DepthDirector +AD | 1.5395 | 0.0167 | 0.8615 | 0.1792 |
On HumanVid:
| Method | Rot. ↓ | Trans. ↓ | Traj. ↑ | Amp. ↑ | F-IFS ↑ |
|---|---|---|---|---|---|
| ReCamMaster | 5.015 | 0.0542 | 0.237 | 0.614 | 0.9298 |
| ReCamMaster +AD | 4.025 | 0.0643 | 0.345 | 0.657 | 0.9317 |
| DepthDirector | 8.112 | 0.0770 | 0.277 | 0.615 | 0.9078 |
| DepthDirector +AD | 4.811 | 0.0710 | 0.328 | 0.640 | 0.9195 |
AD improves eight of the nine reported metrics for each baseline. Both models improve in SpatialVID geometric consistency and HumanVid trajectory, amplitude, and identity fidelity. For example, ReCamMaster's trajectory fidelity increases from 0.237 to 0.345, and DepthDirector's HumanVid rotation error decreases from 8.112 to 4.811.
We further compared RGB + AD with RGB + CAMEO to examine the source of the addressing gains. RGB + AD's translation-error reduction is larger in the lowest-coverage third than in the highest: 41.4% versus 7.6% on HumanVid, and 33.1% versus 15.9% on SpatialVID. SpatialVID geometric gains are also larger at low coverage. The percentages compare group-mean errors, with per-trajectory coverage defined as the median valid-warp fraction over 81 frames. These findings support stronger cross-frame source addressing when the synchronized source view provides limited direct support.
Given the rebuttal timeline, these are straightforward integrations without method-specific hyperparameter tuning. We believe the gains make AD a promising training objective for a broader range of video-retaking architectures and motivate further method-specific optimization.
Expanded in the revised manuscript: Section 4.3; Appendix A.8 / Table 9 for the AD integrations; and Appendix A.5 / Figure 16 for the coverage-dependent RGB + AD / RGB + CAMEO comparison.
IV. Terminology and AD as an auxiliary objective
We have made the ordering explicit at the first introduction of the terms:
- Appearance-First Training: train the appearance expert first, then freeze it to supervise geometry training.
- Geometry-First Retaking: predict target depth first, then use it for RGB synthesis.
- AD: an auxiliary addressing loss used during geometry training, alongside the depth flow-matching loss, with objective .
Expanded in the revised manuscript: Introduction and Sections 3.3–3.4; Appendix B.6 gives the update schedule and reduction details.
Response to Reviewer p7V7 — Part 3/3: Trajectories, Hyperparameters, and Noise Levels
V. Construction and scale normalization of target trajectories
Thank you for asking us to make this protocol explicit. SpatialVID, HumanVid, and DAVIS use the standard trajectories released by ReCamMaster, with the same Pi3X-based scale construction and target cameras supplied to all compared methods.
- Scene radius: from the full 81-frame Pi3X depth sequence, we collect finite positive depths in a central window of at most 200 × 200 pixels per frame and set . A frame with no valid central depth uses its full-image valid depths.
- Translation amplitude: for a standard trajectory expressed relative to its first frame, let be its maximum translation magnitude. Translating paths are scaled by , giving a maximum displacement of 0.3r. Orientations look toward the scene origin. Zero-translation templates use their rotation curves to generate constant-radius orbits.
- Common coordinates: source and target cameras undergo the same rigid transformation so that the first source camera is the identity. Depth and camera translation retain their shared Pi3X reconstruction units within each scene; intrinsics follow the actual resize and crop.
- Per-model reconstruction: when a method reconstructs its own source geometry, scale alignment maps the fixed target cameras into its reconstruction coordinates, preserving the benchmark trajectory definition.
The held-out UE benchmark is constructed separately in the engine:
- We sample navigable scene regions and create multi-person events with fixed actor identities, animations, and event timing.
- Camera paths are planned around actor/region anchors; following paths update their anchors with the actors' actual motion. Candidate seeds, path distances, amplitudes, and curvatures are recorded.
- We check camera/environment and camera/actor collisions using continuous sweeps and a 25 cm camera sphere, together with actor support, collisions, speed, and the intended visibility changes for return trajectories.
- Paired videos replay the same event using frozen per-frame camera positions, rotations, and intrinsics. UE150 selects one pair from each of 150 independent regions, retaining 81 frames, the same initial source/target pose and intrinsics, and continuous landscape motion.
UE camera positions are converted from centimeters to meters, and the camera axes are transformed to the model convention, consistently with metric GT depth. The paths retain their engine-planned motion, while the 0.3r rule applies to the three reconstructed monocular benchmarks.
Expanded in the revised manuscript: Section 4.1; Appendix B.2 for scene scale and standard trajectories; Appendix B.4 / Tables 12–13 and Figure 19 for held-out UE construction.
VI. Selection of AD hyperparameters
We used a simple heuristic strategy to select the AD settings, guided by attention diagnostics on training examples across blocks 15–29 at σ = 1. We chose three blocks with complementary properties: block 17 has the strongest temporal correlation (0.897), block 23 assigns the largest attention mass to source tokens (31.51%), and block 27 has the strongest same-time argmax rate (92.86%). Their losses receive equal weights. We set β = 0.05 based on the initial magnitudes of the KL and flow-matching losses, and apply AD every five optimizer steps to balance supervision frequency and training cost. The reported results use these heuristic settings without extensive hyperparameter tuning; more systematic tuning of the block selection, loss weight, and update frequency may yield further improvements.
Expanded in the revised manuscript: Appendix B.8 / Table 15 for the block-selection analysis and Appendix B.6 for the exact loss and update schedule.
VII. Probing at other noise levels
We added RNT, which evaluates the addressing KL loss at randomly sampled noise levels, while the teacher and student share the sampled level and Gaussian noise. GRAFT uses the pure-noise endpoint.
| Dataset | KL probing | Rot. ↓ | Trans. ↓ |
|---|---|---|---|
| SpatialVID | Random noise level (RNT) | 1.280 | 0.015 |
| SpatialVID | Pure-noise endpoint (GRAFT) | 0.763 | 0.011 |
| HumanVid | Random noise level (RNT) | 3.125 | 0.054 |
| HumanVid | Pure-noise endpoint (GRAFT) | 1.709 | 0.040 |
On HumanVid, pure-noise probing also increases trajectory fidelity from 0.361 to 0.379 and amplitude fidelity from 0.655 to 0.676. These results support the motivation for the pure-noise endpoint: it removes residual target RGB from the teacher's noisy input, allowing its source addressing to be learned under the source observations and known target layout. RNT is presented as a probing-noise diagnostic and is excluded from the ablation rankings and best/second-best markings.
Expanded in the revised manuscript: Section 4.3 / Table 4, variants (f) RNT and (g) GRAFT; Section 3.2 explains the probing rationale.
Response to Reviewer p7V7 — Part 1/3: Baseline Comparisons and Camera Control
Thank you for the detailed and constructive suggestions. They encouraged us to broaden the baseline comparison, evaluate more challenging camera settings, and test AD beyond GRAFT. We appreciate this concrete guidance and the opportunity to strengthen both the experiments and their presentation. Below, we address the main concerns and provide the corresponding results and manuscript references.
I. Vista4D, NeoVerse, and GEN3C comparisons
We added all three methods to the SpatialVID and HumanVid quantitative comparisons and to the qualitative comparisons. We also expanded Related Work to describe how Vista4D and NeoVerse train for robustness to reconstruction artifacts.
The added camera-following and fidelity results are:
SpatialVID FVD here uses I3D features. GRAFT obtains lower rotation and translation errors, FID, and FVD than these three methods on both benchmarks. The full tables also report geometric consistency, subject motion, and identity metrics.
We additionally compare against synchronized target RGB on the official iPhone benchmark and on our held-out UE set. GRAFT obtains the best average rank of 1.385 across the 13 GT-referenced scores, with the full metric-by-metric results in the revised main paper. Qualitative comparisons include human-centric videos, matched GT views in UE, and additional source videos.
Expanded in the revised manuscript: Related Work; Section 4.2 / Tables 1–3; Figure 6; and Appendix A.2.1 / Figures 9–10.
II. Challenging camera trajectories and changing intrinsics
We expanded the camera evaluation in three ways:
The UE trajectory mixture includes 45 dolly/tracking/lateral-translation pairs, 30 S-curve/figure-eight/crane/overhead pairs, 30 large-angle turn-away or sky-reveal return pairs, 15 coupled intrinsic–extrinsic pairs, and 30 additional orbit/reveal/tracking pairs. Average valid-warp coverage is 45.75% overall and 18.13% in the large-angle turn-away-and-return group, providing cases with substantial changes in visibility.
The updated supplementary demo includes 8 complex-trajectory examples, 4 varying-intrinsics examples, and 4 joint intrinsic–extrinsic examples, all with synchronized camera or parameter visualizations. It also presents matched baseline comparisons with Vista4D, NeoVerse, and GEN3C, and will be provided with the supplementary material.
Expanded in the revised manuscript: Appendix A.2.2 / Figure 11 for complex trajectories; Appendix A.2.3 / Figure 12 for changing intrinsics; and Appendix B.4 / Tables 12–13 and Figure 19 for the UE benchmark.