acceptodds
Withdrawn from ICLR 2027

CanoCam: Gimbal and Intrinsics Control for Video Generation via Canonical Views

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Camera-controlled video generation remains unreliable under extreme pitch and roll, wide fields of view, and strong lens distortion. These camera configurations are rare in training data, making their visual effects difficult to learn jointly with scene dynamics. We present CanoCam, which reduces this learning burden through canonical video generation and geometric reprojection. Our approach exploits a geometric distinction: translation induces depth-dependent parallax and visibility changes, whereas rotation and changes in intrinsics admit a depth-independent ray mapping at a fixed camera center. CanoCam first generates canonical videos under simplified camera conditions while preserving the requested camera-center trajectory. We consider two canonical representations: level perspective views that retain yaw and panoramic views that provide full angular coverage. Reprojection then applies the remaining rotation and target intrinsics. For both representations, a target-camera-conditioned video model completes missing regions and refines the reprojected content. Experiments on SpatialVID-Extreme, Prompt Entanglement, and PanShot show improved camera control over strong baselines across extreme orientations and diverse camera models, including both pinhole and fisheye cameras. Controlled ablations validate canonicalization and geometric reprojection as a simple and effective approach to robust camera control. The code will be made publicly available upon acceptance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
camera-controlled video generation, video diffusion models, camera intrinsics, lens distortion

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_xNbPOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces CanoCam, a camera controlled video generation framework for controling both extrinsic and instrinsic. The main idea is that it decompose the target camera into the movement of a canonical camera and residual rotation + instrinsic. The canonical camera part is handled by a level-view or panoramic generator, while the residual rotation and intrinsic reproject and refine the projection our of the generator to get the final output video. The level-view generator is Wan 2.1 with UCPE camera conditioning, and panoramic generator is PanoWan. The reprojection and refinement process is a video outpainting process.The results demonstrates that the proposed method gives better perfomance than existing UCPE and GimbalDiffusion.

Critical strengths
  1. I think the idea of this paper is sound and clear. Presentation is also good. The supplementary includes video introduction, makes this paper more easy to understand.

  2. Results are well-demonstrated and better than exisiting baselines. Supplementary also provides sufficient results.

Critical weaknesses
  1. Given that UCPE already contains instrinsic camera parameter control ability, the reprojection and refinement process seems not necessary, or in another world, if we don't want to control intrinsics in cannonical generation, why use UCPE in this step?

  2. The refinement process seems make the camera less accurate, shown in the supp table 5, which is expected.

  3. The novelty of this paper is not obvious. It's a simple reuse of exisiting camera control model and warp, outpaint it to refine. The idea makes sense but not interesting.

Questions for authors

See weaknesses. No other questions.

1 comment

Thank you for your thoughtful feedback and for finding the idea sound and clear. We address your concerns below.

1. Why reproject when UCPE already controls intrinsics?

(a) Conditioning does not guarantee accuracy. UCPE encodes the full camera geometry, but this does not guarantee that a conditioned generator will reproduce it accurately.

ErrorUCPECanoCam-P
PanShot vertical FoV (°), all 272 clips (Table 2)9.478.34
PanShot vertical FoV (°), 81 known-source-disjoint clips11.037.76
PanShot distortion k₁, 81 known-source-disjoint clips0.1910.129
SpatialVID-Extreme PF pitch (°) (Table 1)16.099.18
SpatialVID-Extreme GC₆₀ pitch (°) (Table 1)14.898.47

On the source-disjoint clips (see our reply to Reviewer xtkZ), the paired differences in vertical FoV and k₁ are significant (95% bootstrap over source videos), as are both SpatialVID-Extreme differences.

Orientation accuracy also depends on viewpoint. On Prompt Entanglement (Table 7), UCPE’s PF pitch error rises from 8.0° at absolute pitches up to 20° to 18.2° at 30° to 50° and 40.9° at 60° to 90°. We therefore generate level canonical views and apply the remaining rotation and lens mapping by same-center reprojection wherever the canonical view covers the target rays.

(b) Why use UCPE conditioning in the canonical stage? We adopt UCPE’s camera-conditioning mechanism and fine-tune both the Wan2.1 backbone and camera adapters for canonical-view generation. In CanoCam-P, the generator still needs to follow the requested camera-center trajectory and yaw, while using fixed intrinsics and zero pitch and roll. UCPE provides these camera conditions. Our framework is not tied to UCPE, as the 5B variant uses GimbalDiffusion’s Plücker conditioning instead (Appendix D).

2. Refinement and camera accuracy

(a) CanoCam-E. Refinement improves the visual quality of projected panoramic views at the cost of some camera accuracy. On SpatialVID-Extreme, the generator produces 576×1152 ERP panoramas covering the full sphere. Each 480×720 target view uses only part of the panorama, so its effective source resolution can be much lower than its output resolution. Reprojection can therefore require substantial upsampling and produce blur and resampling artifacts. We use refinement to restore detail and reduce these artifacts.

Table 5 (Appendix C.1) quantifies the tradeoff. Q-Align improves from 0.634 to 0.719 on SpatialVID-Extreme and from 0.558 to 0.793 on PanShot, while PF pitch error on SpatialVID-Extreme increases from 7.94° to 9.29°. On the new held-out high-pitch subset, PF pitch error increases from 11.99° to 14.17° (see our reply to Reviewer xtkZ). We will present projection-only and refined outputs as two options, prioritizing camera accuracy and visual quality, respectively. We will also add a comparison of the same frames before and after refinement, with enlarged crops.

(b) CanoCam-P. Here refinement completes missing content, as an average of 58.4% of target pixels are unobserved by the canonical view. On SpatialVID-Extreme seed 0, completion reduces PF pitch error from 9.14° to 8.87°, rotation error from 38.47° to 13.07°, and FVD from 1281 to 919.

3. Novelty

The warp-then-inpaint pipeline is common. However, our main contribution is a geometric decomposition of camera control that exploits the generator’s lower errors at canonical viewpoints to improve accuracy at challenging target views.

Table 3 directly tests whether the generator should learn the requested pitch or leave it to geometric reprojection. Both source generators are fine-tuned on the same training clips and evaluated with the same refiner (seed 0). Generating with the requested pitch gives 16.91° PF pitch error. Generating level views and applying the target pitch through reprojection reduces this to 8.87°. This shows the benefit of assigning difficult camera changes to analytic geometry while simplifying the generator’s task.

Both stages are adapted to this task. Without canonical fine-tuning, PF pitch error rises to 11.37°. Replacing our refiner with VACE, a general completion model, gives 13.46° with the same reprojected observations and blending (Table 3).

The approach also transfers to another backbone. CanoCam-5B reduces GC₆₀ pitch error from GimbalDiffusion 5B’s 11.38° to 7.91° over four shared seeds (Appendix D).

More broadly, CameraCtrl introduced camera-conditioning modules, CameraCtrl II emphasized diverse camera-annotated data, UCPE improved camera representations, and AC3D showed that camera motion emerges early during denoising. We address camera control from a different angle: using analytic geometry to simplify what the generator must learn and improve accuracy at challenging viewpoints.

Reviewer_xtkZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CanoCam addresses camera-controlled video generation under extreme orientations and varied intrinsics. It generates canonical views along the requested camera-center trajectory, applies the remaining rotation and intrinsics through same-center reprojection, and uses target-camera-conditioned refinement to complete or enhance the result. Perspective and panoramic variants trade spatial detail for angular coverage. Evaluation covers three benchmarks, component ablations, multiple camera estimators, candidate selection, and a second video backbone.

Critical strengths

The decomposition has a sound geometric basis and supporting experiments. Equation 1 (p. 4) explains when reprojection requires no depth. In the seed-0 ablation, with the refiner fixed, canonical fine-tuning reduces PF pitch error from 11.37° to 8.87°; retaining target pitch instead gives 16.91° (Table 3, p. 8). This supports simplifying the source-generation problem, beyond merely applying a known geometric transformation.

Several control gains survive useful cross-checks. Without candidate selection, CanoCam-P reduces GC₆₀ pitch error from GimbalDiffusion 5B’s 11.52° to 8.47° (Table 1, p. 7). On PanShot’s strict known-source-disjoint subset—81 clips from 13 source videos—ViPE rotation sums are 3.03 for CanoCam-P and 3.90 for CanoCam-E, versus UCPE’s 5.50 radians; both reported reduction intervals exclude zero (Table 8, p. 18). These results support a meaningful benefit beyond the selection metric or identified source overlap.

Critical weaknesses

The main extreme-view benchmark also informed model selection. Appendix B (p. 15) states that checkpoints, canonical FoV, and ERP inference resolution were informed by SpatialVID-Extreme seed 0. Additional seeds on the same requests measure sampling variation after those choices, not held-out generalization. Prompt Entanglement provides separate evidence, but CanoCam-E’s pitch difference from GimbalDiffusion 5B remains uncertain: −2.50°, with a scene-clustered 95% interval of [−5.33°, 0.41°] (§C.1, p. 17). Near-vertical assessment also lacks reliable corroboration from GC (Table 11, p. 19). This does not disprove an advantage or negate the PanShot results, but it leaves the robustness of the extreme-view gains insufficiently established.

The target-camera-conditioning ablation does not isolate the claimed incremental benefit. Supplying canonical cameras to the same trained refiner introduces mismatched inputs (§4.3 and Table 3, p. 8). The resulting degradation demonstrates sensitivity to incorrect conditioning, not superiority over a matched refiner trained without camera inputs. VACE is not reported as such a training-matched control. The complete-pipeline results remain valid evidence, but the stronger claim that these experiments establish the benefit of explicit target-camera conditioning needs this distinction resolved.

Questions for authors

Clarification and targeted validation: Which settings were fixed before inspecting SpatialVID-Extreme seed 0? Can frozen settings be tested on a small set of previously unused extreme-view requests, with an independent orientation check on a high-pitch subset? This would directly address the main generalization concern.

Requested component control: Can you compare against a refiner trained without camera inputs, keeping initialization, observations, data, and training budget matched? This would distinguish the value of explicit camera conditioning from sensitivity to mismatched cameras.

Reanalysis of existing outputs: Could you report lens and absolute-orientation errors, with paired source-clustered intervals, on PanShot’s strict known-source-disjoint subset? Table 8 establishes trajectory-rotation gains there; corresponding intrinsics results would strengthen the evidence for another central claim without requiring new generations.

1 comment

Thank you for your careful review. We added a held-out evaluation, a matched refiner control, and a source-disjoint reanalysis.

1. Model selection and held-out evaluation (W1, Q1)

Appendix B notes that checkpoints, canonical FoV, and ERP resolution were selected using SpatialVID-Extreme seed 0. We therefore tested new requests with the submitted settings fixed and no further tuning.

We constructed 96 new requests from SpatialVID-HQ clips outside SpatialVID-Extreme, using real prompts and camera paths with synthetic rotations. We used 48 requests at moderate pitch (35° to 55°) and 48 at high pitch (65° to 85°). Each method generated one video per request without selection. Requests, analysis protocol, and checkpoints were fixed before generation and will be released.

Moderate pitch: 35° to 55°

MethodPF pitch (°)GC₆₀ pitch (°)VGGT rotation (°)
UCPE14.7417.5913.05
GimbalDiffusion 5B8.5214.9212.90
CanoCam-P7.279.6310.65
CanoCam-E6.5912.059.55
CanoCam-E, projection only5.9811.707.82

High pitch: 65° to 85°

MethodPF pitch (°)VGGT rotation (°)
UCPE31.3316.33
GimbalDiffusion 5B14.8518.96
CanoCam-P23.0514.29
CanoCam-E14.1710.19
CanoCam-E, projection only11.999.30

Lower is better. GC₆₀ uses only frames with |pitch| ≤ 60°, excluding the high-pitch group.

At moderate pitch, CanoCam-P lowers GC₆₀ pitch error by 5.29° versus GimbalDiffusion 5B (paired 95% CI [3.41°, 7.32°]). At high pitch, CanoCam-E has lower mean PF pitch error, 14.17° versus 14.85°, and projection-only CanoCam-E has the lowest mean, 11.99°. All CanoCam variants have lower relative rotation errors than GimbalDiffusion 5B in both groups, with paired 95% intervals excluding zero.

High-pitch estimator validation. To seek an independent check, we compared PF, GeoCalib, and G3T on known-pose images. We geometrically verified the gravity directions of 53 real panoramas using vertical scene lines, then rendered 6,360 images with known pitch angles, with roll fixed at 0°. We tested G3T with both single images and groups of eight views.

At |pitch| ≥ 65°, GeoCalib's mean absolute error exceeded 26°. Even with eight views, G3T underestimated pitch magnitude by approximately 9° to 17°. PF performed better in most tested high-pitch settings, with mean absolute errors of 6.2° to 8.4° at |pitch| = 65° or 75°, and 10.3° to 13.8° at 85°. These tests support retaining PF. The calibration protocol and full results will appear in the revised appendix.

2. Matched refiner control (W2, Q2)

We agree that the original ablation tests sensitivity to camera mismatch and VACE is not training-matched.

We retrained both refiners for this matched comparison. The control receives a fixed neutral camera during both training and inference. This removes target-specific camera information while preserving the architecture. Both refiners share initialization, the 12,006 training clips, cached observations, optimizer, data order, seeds, and the 2,500-step schedule.

Results on SpatialVID-Extreme use the same canonical observations and blending, with 138 requests and seed 0:

Camera conditioningPF pitch (°)GC₆₀ pitch (°)VGGT rotation (°)VGGT translation
Target camera9.468.2313.690.406
Fixed neutral camera11.618.6615.110.431

Target-camera conditioning reduces PF pitch error by 2.14° (95% CI [0.42°, 4.20°]) and rotation error by 1.42° ([0.54°, 2.45°]). An exploratory breakdown shows the largest PF improvement, 6.28°, in the steepest third of requests, whose mean absolute pitch is 58°.

3. Lens and absolute orientation on the strict subset (Q3)

We excluded test clips sharing an original source video with any identified training source, even across different time segments. This leaves 81 clips from 13 source videos in the Table 2 evaluation set. We re-evaluated saved outputs using the same GeoCalib protocol.

MethodvFoV (°)k₁k₂Pitch (°)Roll (°)
UCPE11.030.1910.1375.204.34
GimbalDiffusion UCPE19.110.2580.1296.024.32
CanoCam-P7.76*0.129*0.1274.674.21
CanoCam-E8.17*0.1560.1444.684.50

* Lower error than UCPE with paired 95% bootstrap intervals excluding zero, resampling the 13 source videos (10,000 draws).

CanoCam-P reduces vFoV error by 3.26° (95% CI [1.92°, 4.97°]) and k₁ error by 0.062 ([0.010, 0.140]) relative to UCPE. Against GimbalDiffusion UCPE, both variants have significantly lower vFoV, k₁, and pitch errors.

Here, k₁ and k₂ are GeoCalib-fitted radial coefficients, not the requested UCM ξ. Full paired source-clustered intervals will appear in the revised appendix.

Reviewer_Ja4aOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

CanoCam is a camera-controlled video generation framework for extreme pitch/roll, wide FoV, and strong lens distortion. A generator first produces a canonical video (level perspective or panoramic) along the requested camera-center trajectory. Rotation and intrinsics are then applied by geometric reprojection, and a target-camera-conditioned refiner fills the missing regions. Experiments on SpatialVID-Extreme, Prompt Entanglement, and PanShot show better camera control than the baselines on most metrics.

Critical strengths
  1. Reprojection reflects the target camera exactly in the visible regions.
  2. As a result, the method achieves the best or near-best quantitative performance on most benchmarks.
Critical weaknesses
  1. Lack of novelty. The key idea is similar to [1]. In particular, both use geometric reprojection for the visible regions and a diffusion model for the rest. The paper needs to discuss clearly how it differs from [1].
  2. Non-trivial computational cost and inference memory. One candidate takes about 63.2 B200-seconds for CanoCam-P and about 115 B200-seconds for CanoCam-E, versus 12.3 for GimbalDiffusion 5B. This is roughly 5–9× slower, even though the paper notes it is not an exact matched-compute comparison. Best-of-K selection increases the cost further, in proportion to K. On a GPU as strong as the B200, this is a non-trivial limitation in practice. Since the method uses the generator and the refiner together, VRAM is also expected to be high. The paper should report inference VRAM for CanoCam and the baselines.
  3. Inappropriate qualitative samples. The main qualitative results in Figure 4 are hard to compare against the baselines. The figures are complex and the differences are hard to identify, so the paper's superiority does not come through. It is hard to tell how close "Ours" is to the ground truth.

References [1] Yu et al. TrajectoryCrafter: Redirecting Camera Trajectory for Monocular Videos via Diffusion Models. https://arxiv.org/abs/2503.05638

Questions for authors

Please see weakness,

3 comments

Thank you for your thoughtful feedback. We address your concerns below.

1. Novelty and Differences from TrajectoryCrafter

The warp-then-inpaint pipeline is common. However, our main contribution is demonstrating that this simple strategy can improve camera-control accuracy in video generation.

Compared with TrajectoryCrafter, our method differs in three ways:

(a) Task. Our task is camera-controlled text-to-video generation, whereas TrajectoryCrafter takes a video as input and redirects its camera trajectory.

(b) Warping. Our method uses depth-independent reprojection. We isolate rotation and intrinsic changes, making it a homography for pinhole cameras. The canonical and target cameras share the same center at each frame. This avoids depth-based warping and errors caused by inaccurate depth estimates.

(c) Beyond warping. Warping is a means to improve camera control, not an end in itself. Camera-control errors depend on viewpoint. On our held-out requests, UCPE’s PF pitch error rises from 14.7° at moderate pitch to 31.3° at high pitch. We therefore first generate canonical views, taking advantage of the model’s greater accuracy at moderate viewpoints, then apply the remaining camera changes analytically. This reduces the viewpoint variation the source model must learn and improves camera-control accuracy.

We will add TrajectoryCrafter to the related work and discuss camera-editing methods more broadly.

2. Computational Cost and Inference Memory

We measured all methods on one B200 with matched output settings: 49 frames at 480×720, 50 denoising steps per stage, and guidance scale 5. Runtime excludes initial model loading. Peak process GPU memory was sampled every 200 ms.

MethodBackbonePeak GPU memoryTime per video
UCPEWan2.1-1.3B19.7 GiB30.1 s
GimbalDiffusion UCPEWan2.1-1.3B19.4 GiB30.7 s
GimbalDiffusion 5BWan2.2-TI2V-5B29.6 GiB10.8 s
CanoCam-PWan2.1-1.3B20.0 GiB62.2 s
CanoCam-EWan2.1-1.3B25.4 GiB113.9 s

Memory. CanoCam needs about the same memory as its base model. Its generator and refiner run one after the other, each with UCPE's 1.3B architecture. CanoCam-E needs more because its 576×1152 panorama has 1.9 times the pixels of the target view. Both variants also run on a single 24 GB RTX 4090, where part of our results were generated. For CanoCam-E, we offload the 10.6 GiB T5 text encoder.

Time. On Wan2.1-1.3B, CanoCam-P runs the backbone twice and takes about twice as long as UCPE and GimbalDiffusion UCPE. GimbalDiffusion 5B is faster per pass because the Wan2.2-TI2V-5B VAE compresses 16×16 instead of 8×8, so its transformer processes about a quarter of the tokens (4,485 vs 17,550 per video). On that backbone, CanoCam again costs about two passes (CanoCam-5B, Appendix D) and lowers GC₆₀ pitch error from 11.38° to 7.91° relative to GimbalDiffusion 5B (four shared seeds).

Best-of-K at matched cost. Each baseline gets twice its own cost, or about the time of one CanoCam-P candidate:

MethodCandidates (K)Time per videoGC₆₀ pitch (°)VGGT rotation (°)VGGT translation
CanoCam-P162.2 s8.4713.270.409
UCPE261 s13.5317.720.424†
GimbalDiffusion UCPE262 s13.7112.32†0.474
GimbalDiffusion 5B222 s10.9714.17†0.734
GimbalDiffusion 5B556 s10.2513.64†0.742

Means over the 138 SpatialVID-Extreme requests (GC₆₀: 131). Lower is better. † Not significantly different from CanoCam-P (paired bootstrap, 95%). At matched cost, every baseline keeps a higher GC₆₀ pitch error than CanoCam-P. UCPE also has a higher rotation error, and both GimbalDiffusion models a higher translation error.

Best-of-K is optional and available to every method. Camera estimators are imperfect, so selecting and evaluating candidates with the same estimator can produce gains that partly reflect estimator noise. We therefore use only one candidate per method in all main comparisons (Tables 1 to 3) and report Best-of-K separately as an optional enhancement available to all methods. We will add these measurements to the paper.

3. Qualitative Samples

We agree that Figure 4 should make camera accuracy easier to assess. In the revision we will upload this week, we will use larger frames and fewer methods per row, overlay the requested and estimated horizons and up directions, and report each video’s camera error. We will also include ground-truth frames for PanShot. For SpatialVID-Extreme, which has no ground-truth footage for its synthetic camera requests, we will use the requested camera as the reference.

Thank you for the detailed responses. The memory report and the planned Figure 4 update address my concerns about VRAM and the qualitative samples.

My remaining concern is novelty. Both CanoCam and TrajectoryCrafter [1] reproject the visible regions and complete the rest with a diffusion model, so the task difference does not explain the mechanism difference. Table 3 shows that the decomposition works, but it does not show a methodological advance over warp-then-inpaint. The two-stage design also doubles the inference cost (CanoCam-P is about 2x slower than UCPE: 62.2 s vs. 30.1 s). Given this additional cost, does the proposed second stage outperform TrajectoryCrafter when applied to the same canonical videos?

[1] W. Yu et al., "TrajectoryCrafter: Redirecting Camera Trajectory for Monocular Videos via Diffusion Models," arXiv:2503.05638, 2025.

Thank you. CanoCam-P achieves lower camera errors and higher Q-Align scores on the same canonical videos.

We keep the canonical videos, warps, masks, and paste-back fixed. TrajectoryCrafter uses its released completion model with the first 10 canonical frames as reference, at 384×672 before resizing outputs to 480×720. Both methods use one sample per request.

Results cover SpatialVID-Extreme (138 requests, seed 0) and an additional held-out set (96 requests, CameraBench prompts and synthetic paths). Differences are TrajectoryCrafter minus CanoCam-P, with 95% paired bootstrap intervals.

EvaluationMetricCanoCam-PTrajectoryCrafterPaired difference
SpatialVID-ExtremePF pitch ↓8.87°16.09°+7.21° [4.41, 10.30]
GC₆₀ pitch ↓8.30°11.84°+3.54° [1.61, 5.72]
VGGT rotation ↓13.07°17.46°+4.39° [2.69, 6.32]
Q-Align ↑0.7320.715−0.017 [−0.028, −0.006]
Additional held-out setPF pitch ↓24.39°44.26°+19.87° [14.79, 24.73]
GC₆₀ pitch, moderate group ↓16.28°26.22°+9.93° [6.18, 13.84]
VGGT rotation ↓16.88°29.34°+12.46° [9.62, 15.44]
Q-Align ↑0.7310.686−0.045 [−0.063, −0.027]

On SpatialVID-Extreme, the canonical view leaves 58.4% of target pixels unobserved on average. This requires generating much more of the image than in the examples shown in TrajectoryCrafter’s Fig. 4, where most content remains visible. With so much content to generate, the refiner benefits from knowing where the target camera is pointing. Our matched ablation for Reviewer xtkZ shows this directly: adding target-camera conditioning reduces PF pitch error from 11.61° to 9.46°.

Our core contribution is generating canonical views before target-view completion. This simplifies learned camera control and assigns the remaining rotation and intrinsics to exact, depth-free reprojection. Table 3 isolates this choice: PF pitch error falls from 16.91° to 8.87° with the same refiner. We use inpainting to complete the missing regions and do not claim a new inpainting mechanism.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.