acceptodds
Under review as a conference paper at ICLR 2027

GT2: Directing World Modeling via Geometry-Guided Test-Time Optimization

Abstract

Video world models offer rich priors for simulating dynamic environments. However, jointly controlling camera and object motions remains challenging because their effects are coupled in the observed video and must be composed consistently in a shared 3D coordinate system. Existing methods typically address this challenge through task-specific training on large-scale videos with detailed camera and object motion annotations, making joint control resource-intensive and difficult to adapt. We propose GT2 (Geometry-Guided Test-Time Optimization), a framework that enables joint camera and object control in pretrained video generators by providing explicit geometric supervision at test time, without additional large-scale training. Specifically, GT2 composes prescribed camera motions and object translations in a shared 3D coordinate system and projects the input scene to construct a coarse preview of their combined visual effects. This geometry-aware preview serves as supervision for test-time optimization, directly steering generation toward the prescribed scene dynamics. Experiments show that GT2 improves joint motion controllability while maintaining high visual quality, reducing camera translation error, camera rotation error, and object control error by approximately 28%, 6%, and 10%, respectively, compared with leading baselines.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video generation, joint camera and object control, test-time optimization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_pdsGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

This paper addresses the challenge of jointly controlling camera and object motion in video generation. It proposes GT2, which first composes camera and object motions in a unified coordinate system and generates a coarse preview of their combined visual effects. This preview is then used as supervision to optimize a LoRA module at inference time. The optimized LoRA is applied during the early denoising steps to achieve coordinated control. Experiments show that GT2 improves joint motion controllability while preserving high visual quality.

Critical strengths
  • Camera and object motions are coupled in image space, making joint camera–object control a practically valuable problem to explore. The paper composes both types of control in a unified coordinate system, explicitly handles occlusions, and uses the resulting coarse preview as supervision during the early denoising stages. The overall framework is coherent, and the motivation behind each component is clearly articulated.

  • The proposed method does not rely on large-scale annotated video datasets. Instead, it performs optimization directly at test time, reducing data requirements while improving transferability across different video generation backbones.

  • In the main joint-control experiments, GT2 outperforms all compared methods in RotErr, TransErr, and ObjMC, while achieving an Overall Score close to the best-performing method.

Critical weaknesses
  • Monocular point-cloud projection, Gaussian object cores, masked latent reconstruction, and LoRA-based test-time optimization are all well established in prior work. The main contribution therefore lies in their systematic integration. Moreover, the current method supports only translational object motion. Generating each video requires approximately 612 seconds on two 48GB GPUs, which limits its practical applicability.

  • The conclusions regarding joint control are primarily based on a self-constructed benchmark consisting of only 100 samples manually selected and annotated by the authors.

Questions for authors

Could the authors provide additional joint-control experiments on external datasets and release the complete benchmark manifest?

6 comments

Q1. The components are established; what is the contribution beyond their integration?

A1. Thanks for raising this concern. Our contribution is a geometry-guided test-time optimization framework for joint camera–object control in pretrained video generators without task-specific offline training. We do not claim point-cloud projection, Gaussian object cores, masked reconstruction, or LoRA themselves as new contributions.

This is non-trivial because simply combining projection, masked reconstruction, and LoRA leaves projection-hole artifacts in the latent target and allows background pixels to dominate object supervision. To address this, we complete the preview before latent encoding while excluding filled regions from supervision, and separately normalize reliable background and object losses. These designs address target contamination and unbalanced control supervision within the joint optimization (Secs. 2.1–2.2).

Q2. The current method supports only translational object motion.

A2. Thanks for raising this point. GT2 supports independently prescribed 3D translation trajectories for multiple objects under simultaneous camera movement. Each control specifies the full per-frame path, including depth-direction motion (Appendix C), allowing object motion to be specified independently of viewpoint changes. Our focus is this joint trajectory-control capability; explicit object rotations and non-rigid transformations are not supported by the current interface (Appendix O).

To enable more flexible object motion, we further explore a cut-and-drag extension of our test-time optimization framework. Specifically, we cut out selected regions from the input image and animate them along user-specified trajectories, optionally incorporating in-plane rotation and scaling, to construct a coarse motion preview. We then use this preview and its corresponding foreground and valid-background masks to guide our existing test-time optimization, allowing the generative model to refine the coarse edits into a coherent video. This extension enables direct image-space motion control without requiring 3D reconstruction. We present a qualitative case study with demo videos in the newly submitted supplementary material (cut_drag_demos.mp4) and Appendix.P of the revised paper

Q3. Generating a video takes 612 seconds on two 48 GB GPUs, limiting practical applicability.

A3. Thanks for raising this concern. The same-backbone comparison isolates the additional cost of our control: on the same two-GPU setup, latency increases from 360 s to 612 s and peak memory from 25 GB to 40 GB per GPU (Appendix J, Table 15 in the original version; Tale 18 in the revision). In return, RotErr, TransErr, and ObjMC decrease by approximately 55%, 56%, and 27%, with comparable Overall Scores (Table 2). Thus, the additional 252 s per video improves joint motion control without task-specific offline training.

For system-level context, 81-frame joint generation takes 612 s on two 48 GB GPUs with GT2’s Wan-5B backbone at 1280 × 704, versus over 3600 s on eight 48 GB GPUs with VerseCrafter’s Wan-14B at 1280 × 720 (Table 14 in the original version; Table 17 in the revision). GT2 also has lower control errors (Table 1). These are different-backbone systems, so this comparison describes their reported resource requirements. GT2 targets controllable video synthesis rather than real-time interaction; the reported latency excludes model loading, preview construction, and VAE encoding/decoding (Appendix B).

We further evaluate the inference cost of WorldForge, a training-free camera-only control baseline using the same Wan2.2-5B backbone, under the same two-GPU setup. WorldForge with Wan2.2-5B requires 742 seconds for inference, with a peak memory usage of 35 GB per GPU. In comparison, our method achieves superior camera controllability, as shown in Table 1, while additionally enabling joint camera–object motion control. Moreover, our method reduces inference latency to 612 seconds, at the cost of only a modest increase in peak GPU memory usage (approximately 5 GB per GPU).

Q4. Why only 100 curated samples? Please provide external joint-control experiments and the complete benchmark manifest.

A4. Thanks for raising this concern. Each joint-control sample requires a source image, object masks, reconstructed scene geometry, and full camera and per-object trajectories in one 3D coordinate system. The trajectories are manually specified through our 3D annotation interface, then supplied consistently to all methods (Appendix C). Camera-only and segmentation datasets do not directly provide these paired controls, so expansion requires per-scene geometry and control preparation, rather than simply collecting more images.

Given this preparation effort, we initially curated 100 fully specified cases for a common evaluation protocol, covering three image sources, 26% multi-object scenes, and varied motion magnitudes (Tables 7–8 in the original verson; Table 8–9 in the revision). We also report five generation seeds in Appendix F, Table 10 (Table 11 in the revision). We will expand the benchmark with more independent scenes, object configurations, and joint-motion conditions.

For external camera-control evidence, Appendix H, Table 12(Table 14 in the revision) evaluates 100 Sekai videos. The requested external joint-control evaluation is a separate experiment. Specifically, we further conduct an experiment on external DAVIS-test dataset, which contains much more complicated object motions and camera motions. The results are shown below. Our method still achieves superior performance.

MethodTransErr ↓RotErr ↓ObjMC ↓
Symphomotion0.4891.3515.367
VerseCrafter0.2460.9415.248
GT20.2260.9224.898

For reproducibility, we will release the complete benchmark manifest with sample identifiers and image sources, prompts, masks, camera/object trajectories, and generation/evaluation configurations.

Dear Reviewer,

As the discussion phase is approaching its end, we hope that the above clarifications and additional experiments have addressed your concerns. If you find our responses satisfactory, we would greatly appreciate your consideration of updating the score to reflect the additional results and discussion. We remain happy to clarify any remaining points during the discussion phase.

Thank you again for your time and consideration.

Reviewer_Sdv3OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents GT2, a geometry-guided test-time optimization framework for joint camera and object motion control in video generation. The method first constructs a coarse video preview by projecting the input scene under prescribed camera and object motions in a shared 3D coordinate system. This preview is then used as supervision to optimize LoRA parameters during the early denoising steps of a pretrained video diffusion model. By combining geometric guidance with the pretrained generation prior, GT2 achieves joint motion control without additional large-scale training. Experiments demonstrate improved camera and object control accuracy while maintaining competitive visual quality.

Critical strengths
  1. The paper addresses a relevant problem in controllable video generation. Unlike existing methods that require task-specific training on annotated videos, GT2 directly adapts a pretrained video generator at test time. This provides a flexible alternative to training-based approaches and makes the method potentially applicable to different video generation backbones.
  2. GT2 achieves competitive results against existing camera and object control methods, with improvements in both control accuracy and video quality metrics. The authors also provide ablations on optimization steps, geometric preprocessing, and supervision components, along with evaluations on additional video backbones. These experiments support the effectiveness of the proposed framework.
Critical weaknesses
  1. The core approach combines geometric projection, masked reconstruction supervision, and LoRA-based test-time optimization. These components are closely related to existing geometry-guided generation and test-time optimization techniques. While the paper compares against direct latent replacement, it does not compare against gradient-based latent optimization using the same geometric supervision. This makes it difficult to determine whether optimizing LoRA parameters offers a meaningful advantage over simpler test-time guidance strategies.
  2. The main benchmark contains only 100 curated samples, with 74% involving a single controlled object and camera rotations limited to 30 degrees. Object control is also restricted primarily to 3D translations. The appendix shows that performance degrades under larger camera rotations, particularly for a full-circle orbit. These limitations raise questions about the robustness and generalizability of the proposed method beyond relatively constrained motion settings.
  3. I could not find accessible video demonstrations or links to supplementary videos in the manuscript. The qualitative comparisons are presented only as selected frames, which are insufficient to assess motion smoothness, temporal consistency, object deformation, and geometric stability throughout the generated sequences. Since the paper's main contribution concerns joint camera and object motion control, direct video comparisons are important for validating its qualitative claims. This limits my confidence in the reported generation quality.
Questions for authors
  1. Can the authors compare GT2 with gradient-based latent optimization using the same geometric preview and supervision loss? A comparison under a similar inference budget would help establish the advantage of LoRA-based optimization.
  2. Can the authors provide video demonstrations of the generated results, including comparisons with the main baselines? If supplementary videos have already been submitted separately, please clarify where they can be accessed. This would help assess temporal consistency and motion quality, which are difficult to evaluate from static frames.
  3. How does control accuracy change as camera and object motions become more challenging? In particular, could the authors provide quantitative results for larger camera rotations and more complex object motions? This would help clarify the practical limitations of the method.
9 comments

Q1. What is new beyond existing geometry-guided generation and test-time optimization methods?

A1. Thanks for raising this concern. Our contribution is to enable joint 3D camera–object control in pretrained video generators through geometry-guided test-time optimization, without task-specific offline training.

This is non-trivial because camera and object motions are coupled in image space, while a preview projected from one image is incomplete and contains warping artifacts. To address this, GT2 composes both controls into a shared 3D geometric target and guides generation through reliable background and object regions, leaving unobserved content to the pretrained generative prior (Secs. 2.1–2.2).

The distinction from TTOM and Zo3T is the control target: TTOM constrains attention layouts and Zo3T constrains features within object boxes, whereas GT2 supervises the combined visual effects of prescribed camera and object motions (Sec. 4). Table 5 (Table 6 in the revision) supports the complementary supervision: removing the background loss raises RotErr from 0.091 to 0.199, while removing the object loss raises ObjMC from 1.574 to 2.176.

Q2. How does LoRA compare with gradient-based latent optimization using the same geometric preview and supervision?

A2. Thanks for suggesting this comparison. We further conduct an experiment comparing both approaches using the same geometric preview, supervision loss, and optimization settings. The alternative updates the noisy latent ztz_t with the generator frozen, while GT2 updates LoRA parameters. This directly compares the optimization variable under shared supervision.

MethodOverall ↑TransErr ↓RotErr ↓ObjMC ↓Mem./GPUTime
Latent opt.87.560.0380.1181.75632 GB568 s
GT2 (LoRA)87.800.0320.0911.57440 GB612 s

GT2 reduces TransErr, RotErr, and ObjMC by 15.8%, 22.9%, and 10.4%, respectively, with similar Overall Scores (87.80 vs. 87.56). It uses 8 GB more peak memory per GPU and takes 7.7% longer. Thus, the observed advantage of LoRA is stronger motion control under this setting, rather than an improvement in overall visual quality.

Q3. How robust is GT2 beyond the 100-sample benchmark, including multiple objects, larger camera rotations, and more complex object motions?

A3. Thanks for raising this concern. We further conduct a stress test with doubled motion magnitudes, where GT2 retains its relative control advantage. To further assess generalization beyond our curated benchmark, we additionally evaluate on the external DAVIS test set under the joint camera–object motion control setting. We report these results below, alongside the existing evidence on scene coverage, different backbones, and large viewpoint changes.

Larger motions. We double the prescribed object displacement vectors, camera-translation magnitude parameters, and camera-rotation angles, retaining the input cases, motion directions, frame counts, and scene reference distance. A 30° prescribed pan becomes 60°, for example. Camera trajectories are regenerated from the doubled parameters.

MethodOverall ↑TransErr ↓RotErr ↓ObjMC ↓
Symphomotion85.940.0680.1453.028
VerseCrafter86.130.0620.1332.580
GT286.460.0590.1132.211

GT2 achieves the lowest errors on all three control metrics and the highest Overall Score in this harder setting. Its errors increase from the original TransErr/RotErr/ObjMC of 0.032/0.091/1.574 (Table 1) to 0.059/0.113/2.211, so the result establishes a retained relative advantage as the motions become more challenging.

Scene and backbone coverage. The 100 cases require manually prescribed camera and per-object trajectories in a shared 3D scene (Appendix C). They cover photographs, movie frames, and AI-generated images, with 26% multi-object scenes, object displacements of 0.8–10.4 m, and camera-center travel of 0–5.0 m (Tables 7–8 in the original version; Tables 8–9 in the revision). Beyond Wan2.2, Appendix I, Table 13 (Table 16 in the revision) applies GT2 to CogVideoX-5B and SkyReels-V2-1.3B, reducing all three control errors relative to each unadapted backbone. These results support transfer across the evaluated generators; the supported object transformations are clarified in A4.

Large viewpoint changes. Appendix N, Fig. 10 evaluates 90°, 120°, 180°, and 360° camera orbits. Performance degrades with larger orbits, with the full-circle example deviating from the prescribed motion as previously unseen surfaces become visible. For in-place yaw, the benchmark’s estimated horizontal FoV (mean 55.56°, standard deviation 12.63°, median 53.88°) also makes a 30° turn a meaningful view-overlap challenge. Orbits additionally involve parallax and disocclusion. The stress test therefore expands the tested motion range without establishing reliable full-circle control.

Generalization to External Real-World Data. To further evaluate the generalization of our method beyond our curated benchmark, we conduct additional experiments on the DAVIS test set, which contains diverse real-world scenes with complex object and camera motions. As shown in the table below, our method consistently outperforms the baselines across all three motion-control metrics, further demonstrating its effectiveness and generalizability in joint camera–object motion control.

MethodTransErr ↓RotErr ↓ObjMC ↓
Symphomotion0.4891.3515.367
VerseCrafter0.2460.9415.248
GT20.2260.9224.898

Q4. Object control is restricted to 3D translations.

A4. Thanks for raising this point. GT2 independently controls each object’s full per-frame 3D translation trajectory, including depth-direction motion, while the camera follows its prescribed trajectory (Appendix C). Specifying the intermediate 3D positions enables coordinated camera and multi-object motion throughout the sequence. The current interface controls object translation; general object rotation and non-rigid transformation are outside its scope, as discussed in Appendix O.

To enable more flexible object motion, we further explore a cut-and-drag extension of our test-time optimization framework. Specifically, we cut out selected regions from the input image and animate them along user-specified trajectories, optionally incorporating in-plane rotation and scaling, to construct a coarse motion preview. We then use this preview and its corresponding foreground and valid-background masks to guide our existing test-time optimization, allowing the generative model to refine the coarse edits into a coherent video. This extension enables a direct image-space motion control without requiring 3D reconstruction. We present a qualitative case study with demo videos in the newly submitted supplementary material (cut_drag_demos.mp4) and Appendix.P of the revised paper.

Q5. Please provide full video comparisons or clarify where the supplementary videos can be accessed.

A5. Thanks for your suggestion! We provide video comparisons with baselines in the newly submitted supplementary material (comparison_videos.mp4).

Reviewer_1kdpOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

GT² builds a 3D preview from single-image depth and object segmentation, specifies camera motion and object translations, and guides video generation using reliable regions. It optimizes LoRA only during early denoising, then disables the adapters and resumes standard generation. It avoids large-scale offline controller training but still optimizes model parameters at test time.

Critical strengths
  1. With the same Wan2.2 backbone, camera and object-motion errors decrease while overall visual quality remains similar. Comparison with direct latent replacement also supports the design's practical value (Tables 2–3).
  2. External Sekai videos, additional generator backbones, and human preferences provide complementary support beyond a single model or metric (Appendices F, H, I, and M).
Critical weaknesses
  1. The need to change model parameters at test time is not fully explained. Existing comparisons support geometric supervision and show an advantage over direct latent replacement. They do not establish why LoRA is preferable to optimizing the latent or noise under the same preview target. Given its extra time and memory costs, this matters to the design's value (§2.2; Tables 2–3 and 15).
  2. Evidence for controlling displacement magnitude is less direct. Camera evaluation first fits an optimal scale, so the error shows agreement after alignment without independently establishing the requested physical distance. Object-motion metrics also combine depth and camera poses from different models; how their scales are reconciled needs explanation (Appendix D).
  3. Motion errors across models are not straightforward to compare. Output lengths differ, while the metrics measure motion between consecutive frames. Without a common time interval, some numerical differences may reflect that choice. Comparisons with the same backbone or output length remain valuable, but the other comparisons need clearer interpretation (Appendices D–E).
Questions for authors
  1. With the same 3D preview, why optimize LoRA instead of the latent or noise? What specific benefit justifies the extra cost?
  2. After optimal scale alignment, how is correct trajectory shape distinguished from correct travel distance? How are object and camera scales reconciled?
  3. Are videos of different lengths compared at the same time interval, and how does this affect conclusions across models?
3 comments

Q1. With the same geometric preview, why optimize LoRA instead of the latent or noise, and what benefit justifies the extra cost?

A1. Thanks for raising this concern. We further conduct an experiment comparing LoRA adaptation with gradient-based latent optimization under the same geometric supervision and optimization settings. The alternative optimizes the noisy latent ztz_t while keeping all model parameters frozen. As shown below, LoRA achieves lower camera and object control errors.

MethodOverall ↑TransErr ↓RotErr ↓ObjMC ↓Mem./GPUTime
Latent opt.87.560.0380.1181.75632 GB568 s
GT2 (LoRA)87.800.0320.0911.57440 GB612 s

LoRA reduces TransErr, RotErr, and ObjMC by 15.8%, 22.9%, and 10.4%, respectively, with similar Overall Scores (87.80 vs. 87.56). The cost is 8 GB more peak memory per GPU and 7.7% longer inference. This comparison supports choosing LoRA when motion-control accuracy is the priority.

Q2. Does scale-aligned evaluation establish displacement magnitude, and how are camera and object scales reconciled?

A2. Thanks for raising this concern. Camera and object evaluation use the same MoGe-v2 reference geometry. We clarify what scale alignment measures and provide a fixed-scale test below.

Camera-motion evaluation. Appendix D.1 aligns the VIPE-estimated camera trajectory to the prescribed trajectory using a global Sim(3) transformation. Aligned TransErr measures consecutive relative translations after fitting one global scale; it does not independently establish absolute travel distance. A trajectory that differs only by a global scale can therefore have a low aligned error.

To separate GT2’s accuracy from the benefit of scale fitting, we fix the scale factor to 1 and retain the remaining evaluation procedure. TransErr of GT2 changes from 0.03236 to 0.03299, an increase of 1.95%. Thus, GT2’s low translation error on these clips is not primarily driven by fitting the global scale,

Object-motion evaluation. Prescribed object trajectories are defined in the initial-frame MoGe-v2 geometry. Sim(3) alignment brings the VIPE camera trajectory into this reference frame and scale. For each generated frame, we back-project object points using MoGe-v2 depth and intrinsics at their predicted metric scale, then transform them with the aligned camera poses. Camera translations and object depths are therefore combined under the same nominal metric convention.

Specifically, with aligned world-to-camera pose (Rs,ts)(R_s,t_s), depth Ds(u)D_s(u), intrinsics KsK_s, and homogeneous pixel coordinate uˉ\bar{u}, the reconstructed world-space point is

Xsworld=Rs⊤(Ds(u)Ks−1uˉ−ts).X_s^{\mathrm{world}} = R_s^\top \left(D_s(u)K_s^{-1}\bar{u} - t_s\right).

ObjMC compares the trajectories of the fitted object Gaussian means with the prescribed trajectories in this reference frame (Appendix D.4). All methods use the same reconstruction and evaluation pipeline, with the usual sensitivity to depth and camera-pose estimation errors.

Q3. Are videos with different output lengths evaluated at the same time interval, and how does this affect cross-model conclusions?

A3. Thanks for raising this concern. Videos with different output lengths are not evaluated at a common adjacent-frame interval. Temporal resampling preserves the prescribed overall trajectory, but each frame-to-frame error then covers a different fraction of that trajectory. Those errors therefore cannot be interpreted as measurements over the same temporal interval.

This frame-count difference does not affect the same-backbone ablation in Table 2 or the 81-frame joint-control comparisons with VerseCrafter and Symphomotion (Table 9 in the original version; Table 10 in the revised version). We will distinguish these matched-length results from comparisons across different output lengths in the revised manuscript, so that the control conclusions are grounded in the corresponding evaluation conditions.

Reviewer_1kdpOpenReview ↗

Reply to the rebuttal

Thank you for the additional experiments and clarifications, which address most of my concerns and make the revised paper more complete.

However, frame-to-frame motion errors are still measured over different temporal intervals for videos of different lengths, so these comparisons remain difficult to interpret. This concern remains despite the clarified discussion.

I will maintain my score.

Thank you for following up on this point. As discussed in Appendix E of the revised manuscript, the original evaluation samples the prescribed trajectory at each model’s native output length. RotErr and TransErr therefore measure trajectory-following accuracy over each model’s native frame-to-frame intervals. For models with different output lengths, these intervals span different fractions of the prescribed trajectory, which should be taken into account when interpreting the scores.

All ablation comparisons use matched frame counts, so the motion errors are evaluated over the same temporal intervals within each comparison. Our main baseline comparisons also use matched 81-frame trajectories, including the joint-control comparisons with SymphoMotion and VerseCrafter (Table 10 of the revised manuscript) and the camera-only comparison with WorldForge.

To directly address the concern, we additionally evaluate the shorter-output baselines on a common 81-frame temporal grid. After applying the same Sim(3) alignment as in the original evaluation, we resample the estimated camera poses using linear interpolation for translation and SLERP for rotation. We then recompute both errors against the original 81-frame prescribed trajectory. This evaluates the motion recoverable from each model’s estimated poses over the same normalized temporal intervals. As shown below, under the common-grid protocol, our method still achieves lower rotation and translation errors than the corresponding shorter-output baselines in both the camera-only and joint-control settings. We will include the common-grid results in the revised manuscript.

Method81-frame TransErr81-frame RotErr
CameraCtrl0.0480.171
ViewCrafter0.0270.100
Ours(Camera-only)0.0230.081
PAC0.0480.137
Ours(joint)0.0320.091

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.