acceptodds
Withdrawn from ICLR 2027

Camera-Controlled Video Generation at Scale

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Camera-controlled video generation has become a promising interface for cinematic content creation, but scaling up 3D-camera-pose-conditioned training remains challenging. In large video corpora, camera translation is typically not aligned to a metric scale, making global motion control ambiguous across examples. Metric-alignment approaches address this issue with a preprocessing pipeline that combines structure-from-motion and metric depth estimation to approximate a shared scale. Applying this pipeline to every training video makes metric alignment a computational bottleneck as the corpus grows, while depth-estimation errors can also affect the aligned scale. To remove this bottleneck, we propose a scale-decoupled formulation for camera conditioning that treats trajectory magnitude as an explicit control variable rather than a quantity that must be metrically calibrated in advance. We normalize translation by its total path length, retaining zero translation for stationary cameras. This preserves direction and relative frame-to-frame magnitude while discarding unreliable global scale. Our goal is scalable training without metric-scale alignment, rather than resolving monocular scale ambiguity. We then inject camera-pose and scale , where controls the overall strength of camera translation, denotes a sinusoidal-style embedding, and denotes Pl\"ucker embedding. The two branches are encoded separately and fed to the diffusion transformer. In addition, to avoid requiring human-annotated trajectory preferences, we introduce trajectory preference optimization (TPO), a preference-learning scheme that keeps the video fixed and constructs negative examples by changing the conditioning trajectory, thereby isolating trajectory fidelity from video quality. After supervised fine-tuning (SFT), our 2B and 5B models reduce translation error by 49.4% and 36.8%, rotation error by 38.2% and 38.9%, FVD by 40.7% and 43.3%, and FID by 15.9% and 12.0%, respectively, relative to size-matched AC3D baselines. At the same total optimization-step count, SFT followed by TPO reduces the 2B model's translation and rotation errors by 4.4% and 4.0%, respectively, relative to SFT alone, while keeping FVD, FID, and CLIP score broadly stable.

Primary area:
generative models
Keywords:
Diffusion Models, Text-to-Video Generation, Camera-Controlled Video Generation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_3YcTOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a scale-decoupled camera representation for camera-controlled text-to-video generation. It introduces two branches to encode camera trajectories and magnitudes. It also uses Trajectory Preference Optimization (TPO) (similar to Condition Preference Optimization) for post-training. The method is evaluated using a dataset assembled from multiple public video sources.

Critical strengths

The paper targets a meaningful and practical problem. The scales are ambiguious in camera-controlled video generation, especially for the Text to Video scenarios. Constructing preference pairs seems meaningful for post-training.

Critical weaknesses
  1. Incomplete manuscripts. It is mentioned in the paper that "Our Wan2.1 version is still training, and its results will be updated." This leaves part of the experimental discussion provisional.
  2. Insufficient differentiation or discussion, compared to CPO [1]. CPO (condition preference optimization) formulates preference learning over alternative control conditions (without cosntraining what formats of the conditions). Therefore, TPO seems like an application of CPO for the camera-conditioned video generation. However, CPO is very briefly cited but the proposed equations share very similar stuctures to CPO.
  3. Insufficiently controlled comparisons and unclear attribution of improvements. The comparisons involve different training datasets and backbones, while the training budgets and numbers of trainable parameters are not sufficiently documented to establish matched conditions. Especially, UCPE is a light trainable adapter and does not train on the proposed dataset. Moreover, UCPE achieves a translation error of 0.0104, compared with 0.0131 for the best proposed variant, where the translation measurements are closely realted to the aims of this paper.
Questions for authors

Please see the weakness section.

  1. More qualitative results are desired.
  2. Please also report the training configurations, including used datasets and trainable parameters of baselines. Moreover, more recent baselines, such as those from UCPE, could also be compared for better understanding.
Reviewer_sCj7OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a method for camera-controllable video generation designed for large-scale training. It decouples camera pose from motion scale through total path length normalization. Additionally, the paper introduces Trajectory Preference Optimization (TPO), which constructs preference pairs by perturbing camera conditions. The authors curated a dataset of approximately 532,000 videos with camera annotations and validated the approach using CogVideoX-2B/5B. Experimental results show improved camera control accuracy and video quality compared to AC3D.

Critical strengths

Camera control is an important task for video generation. The proposed method avoids the need for metric-scale alignment during training, making it easier to leverage camera annotations. It also achieves better results compared to AC3D.

Critical weaknesses
  1. While works such as CamCo, Cavia, and ReRoPE have already incorporated trajectory-scale normalization into camera control training, this paper does not discuss how its approach differs from these existing methods or compare with them. Another key component, TPO, achieves a marginal improvement in trajectory error—approximately 4%—and has been validated solely on a 2B-parameter model; consequently, the stability and scalability of its benefits remain unproven.

  2. The experimental and baseline comparisons are insufficient. The study is limited to CogVideoX, lacking validation on other prominent backbones such as Wan. Moreover, the absence of related work comparison—such as MotionCtrl, GEN3C, Lyra-2—under comparable settings weakens the method's competitiveness. Besides, the current materials lack video demonstrations/comparisons, making it difficult to evaluate temporal consistency, motion smoothness, and the actual effectiveness of the camera control.

Questions for authors
  1. How does your trajectory normalization differ from CamCo, Cavia, and ReRoPE? Discussion or ablations are helpful.

  2. Are TPO’s approximately 4% improvements consistent across different model / eval set, and are results available for the 5B model?

  3. Are results available for your method on Wan or CogvideoX against GEN3C or Lyra-2 under comparable settings?

  4. Could you provide video demonstrations or comparisons to assess temporal consistency, motion smoothness, and camera control accuracy?

Reviewer_BHKfOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The authors attach a camera-control branch to CogVideoX-2B/5B and train it on a multi-source video corpus (~532K clips). Unlike prior work, the method does not rescale camera trajectories to metric units. Instead, each trajectory is normalized to a total path length of 1, and the overall amount of motion is controlled by a separate scalar a. The model is then post-trained with Trajectory Preference Optimization (TPO), which teaches it to prefer the ground-truth trajectory over perturbed trajectories for the same video. Experiments show that the proposed method reduces trajectory errors, FVD, and FID by large margins compared with AC3D and CameraCtrl (Tab. 1), although its translation error is worse than that of UCPE.

Critical strengths
  • The representation is simple. The trajectory normalization and log-scale encoding (Eqs. 6 and 7) are easy to implement.
  • TPO requires neither human labels nor generated videos. Because both sides of each preference pair share the same video and differ only in the trajectory.
Critical weaknesses
  • The paper's main claim, that scale decoupling can replace metric rescaling, is not tested. There is no ablation on the same ~532K data comparing (a) raw camera trajectories, (b) metric trajectories divided by a constant, (c) max-length normalization, and (d) the proposed method with and without the a branch.
  • The authors claim that prior methods (RealCam-I2V, AC3D) rely on depth estimation to recover metric scale, and that this preprocessing creates a computational bottleneck (Sec. 1). However, the proposed pipeline also relies on MegaSAM, which estimates both camera trajectories and depth (Sec. 4). The paper does not report the cost of either pipeline, so the claimed saving is not demonstrated.
  • The contribution of TPO is small(Tab. 1), and a is fixed to 1 during TPO training (App. F).
  • The submission includes unfinished experiments: "Our Wan2.1 version is still training" (Sec. 5.1), and the I2V extension is also still under training (App. J). Since the scale-response experiments (Fig. 4) use only the SFT model, it is unclear whether TPO preserves, or even degrades, motion-strength control at a ≠ 1.
Questions for authors
  • On the same ~532K data, how does the proposed method compare with (a) raw camera trajectories, (b) metric trajectories divided by a constant, and (c) max-length normalization, both with and without the a branch?
  • What is the preprocessing cost (e.g., GPU-hours per 1K clips) of metric alignment compared with your MegaSAM annotation? Without this comparison, how is the claimed bottleneck supported?
  • Does TPO improve trajectory accuracy at a ≠ 1?
  • How does TPO compare with camera CFG, i.e., classifier-free guidance on the camera condition, as used in AC3D? Was camera CFG used at inference?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.