- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces CanoCam, a camera controlled video generation framework for controling both extrinsic and instrinsic. The main idea is that it decompose the target camera into the movement of a canonical camera and residual rotation + instrinsic. The canonical camera part is handled by a level-view or panoramic generator, while the residual rotation and intrinsic reproject and refine the projection our of the generator to get the final output video. The level-view generator is Wan 2.1 with UCPE camera conditioning, and panoramic generator is PanoWan. The reprojection and refinement process is a video outpainting process.The results demonstrates that the proposed method gives better perfomance than existing UCPE and GimbalDiffusion.
Critical strengths
-
I think the idea of this paper is sound and clear. Presentation is also good. The supplementary includes video introduction, makes this paper more easy to understand.
-
Results are well-demonstrated and better than exisiting baselines. Supplementary also provides sufficient results.
Critical weaknesses
-
Given that UCPE already contains instrinsic camera parameter control ability, the reprojection and refinement process seems not necessary, or in another world, if we don't want to control intrinsics in cannonical generation, why use UCPE in this step?
-
The refinement process seems make the camera less accurate, shown in the supp table 5, which is expected.
-
The novelty of this paper is not obvious. It's a simple reuse of exisiting camera control model and warp, outpaint it to refine. The idea makes sense but not interesting.
Questions for authors
See weaknesses. No other questions.
Thank you for your thoughtful feedback and for finding the idea sound and clear. We address your concerns below.
1. Why reproject when UCPE already controls intrinsics?
(a) Conditioning does not guarantee accuracy. UCPE encodes the full camera geometry, but this does not guarantee that a conditioned generator will reproduce it accurately.
On the source-disjoint clips (see our reply to Reviewer xtkZ), the paired differences in vertical FoV and k₁ are significant (95% bootstrap over source videos), as are both SpatialVID-Extreme differences.
Orientation accuracy also depends on viewpoint. On Prompt Entanglement (Table 7), UCPE’s PF pitch error rises from 8.0° at absolute pitches up to 20° to 18.2° at 30° to 50° and 40.9° at 60° to 90°. We therefore generate level canonical views and apply the remaining rotation and lens mapping by same-center reprojection wherever the canonical view covers the target rays.
(b) Why use UCPE conditioning in the canonical stage? We adopt UCPE’s camera-conditioning mechanism and fine-tune both the Wan2.1 backbone and camera adapters for canonical-view generation. In CanoCam-P, the generator still needs to follow the requested camera-center trajectory and yaw, while using fixed intrinsics and zero pitch and roll. UCPE provides these camera conditions. Our framework is not tied to UCPE, as the 5B variant uses GimbalDiffusion’s Plücker conditioning instead (Appendix D).
2. Refinement and camera accuracy
(a) CanoCam-E. Refinement improves the visual quality of projected panoramic views at the cost of some camera accuracy. On SpatialVID-Extreme, the generator produces 576×1152 ERP panoramas covering the full sphere. Each 480×720 target view uses only part of the panorama, so its effective source resolution can be much lower than its output resolution. Reprojection can therefore require substantial upsampling and produce blur and resampling artifacts. We use refinement to restore detail and reduce these artifacts.
Table 5 (Appendix C.1) quantifies the tradeoff. Q-Align improves from 0.634 to 0.719 on SpatialVID-Extreme and from 0.558 to 0.793 on PanShot, while PF pitch error on SpatialVID-Extreme increases from 7.94° to 9.29°. On the new held-out high-pitch subset, PF pitch error increases from 11.99° to 14.17° (see our reply to Reviewer xtkZ). We will present projection-only and refined outputs as two options, prioritizing camera accuracy and visual quality, respectively. We will also add a comparison of the same frames before and after refinement, with enlarged crops.
(b) CanoCam-P. Here refinement completes missing content, as an average of 58.4% of target pixels are unobserved by the canonical view. On SpatialVID-Extreme seed 0, completion reduces PF pitch error from 9.14° to 8.87°, rotation error from 38.47° to 13.07°, and FVD from 1281 to 919.
3. Novelty
The warp-then-inpaint pipeline is common. However, our main contribution is a geometric decomposition of camera control that exploits the generator’s lower errors at canonical viewpoints to improve accuracy at challenging target views.
Table 3 directly tests whether the generator should learn the requested pitch or leave it to geometric reprojection. Both source generators are fine-tuned on the same training clips and evaluated with the same refiner (seed 0). Generating with the requested pitch gives 16.91° PF pitch error. Generating level views and applying the target pitch through reprojection reduces this to 8.87°. This shows the benefit of assigning difficult camera changes to analytic geometry while simplifying the generator’s task.
Both stages are adapted to this task. Without canonical fine-tuning, PF pitch error rises to 11.37°. Replacing our refiner with VACE, a general completion model, gives 13.46° with the same reprojected observations and blending (Table 3).
The approach also transfers to another backbone. CanoCam-5B reduces GC₆₀ pitch error from GimbalDiffusion 5B’s 11.38° to 7.91° over four shared seeds (Appendix D).
More broadly, CameraCtrl introduced camera-conditioning modules, CameraCtrl II emphasized diverse camera-annotated data, UCPE improved camera representations, and AC3D showed that camera motion emerges early during denoising. We address camera control from a different angle: using analytic geometry to simplify what the generator must learn and improve accuracy at challenging viewpoints.