acceptodds
Withdrawn from ICLR 2027

LIFT: Layout-In-Future Video Generation under Large Viewpoint Change via On-Policy Self-Distillation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

10 views

Abstract

We introduce LIFT, a unified image-to-video generation framework that complements camera control with Layout-In-Future control, enabling users to specify what should appear in a future view and where it should appear. This addresses a practical need in controllable video generation: given an initial image, users often care not only about how the camera moves, but also about what the scene should look like at key future moments, especially the final frame. Existing camera controls specify viewpoint trajectories, while text prompts provide only coarse semantic guidance; neither precisely determines the content and spatial layout of future views. This limitation becomes particularly pronounced under large viewpoint changes, where the camera reveals regions that are not visible in the initial image. LIFT therefore uses the last-frame layout as an explicit and intuitive control signal for specifying the desired future scene. Since learning from such sparse layout guidance is substantially more challenging than conditioning on dense per-frame layouts, we adopt on-policy self-distillation (OPSD) to transfer the control capability of a dense-layout teacher to a last-frame-layout student. We further curate LIFT-Vista, a dataset featuring large viewpoint changes with camera and temporally consistent layout annotations. Experiments show that LIFT improves video quality, future-layout controllability, and camera controllability over other methods.

Primary area:
generative models
Keywords:
Video Generation, Layout Control

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_bjEyOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This work presents LIFT, enhancing camera-controlled video generation with future layout control to let users define objects and their positions in future frames. It targets a practically meaningful problem in controllable video generation. To ease training, on-policy self-distillation is adopted, and the LIFT-Vista dataset is built. Experiments verify that LIFT outperforms baselines in video quality, future-layout controllability, and camera controllability.

Critical strengths
  • The problem is practically valuable. Camera trajectory control alone cannot specify the content of newly revealed regions, while last-frame layout conditioning provides an intuitive and low-burden complementary control signal.

  • The method is clear and unified. It jointly represents object locations and semantics using pixel-aligned color-coded layout maps and local text prompts, and its three-stage training procedure follows a clear progression.

  • Experimental results show that LIFT substantially outperforms existing camera-control and object-motion methods on layout-related metrics while maintaining good camera accuracy and video quality.

Critical weaknesses
  • The methodological novelty is somewhat limited. The main components, including Plücker-ray camera conditioning, color-coded layout maps with local prompts, and OPSD, are largely adapted from existing techniques. The primary contribution appears to lie in formulating the future-layout control task and integrating these components, rather than introducing a fundamentally new modeling or learning mechanism.

  • The evaluation setting is relatively restricted. Experiments use short, low-resolution clips (81 frames at 352$\times$640) and appear dominated by static scene-exploration videos. Generalization to longer videos, dynamic scenes, crowded environments, and complex occlusions remains unclear.

Questions for authors
  • Given the same random seed, first frame, and camera trajectory, how does the generated result change when objects in the last-frame layout are moved, replaced, or removed?
Reviewer_hhBJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the issue where new regions—revealed following significant camera movement—often appear blank or lack definable content. To resolve this, the authors propose a method for controllable content generation based on bounding-box (box) constraints, further enhancing control accuracy through an "on-policy self-distillation" technique. However, I find the definition of this downstream task to be fundamentally questionable. In most existing approaches to camera-motion-based video generation, the model is not expected to synthesize contextually coherent content for newly revealed areas; rather, the focus is on maintaining geometric consistency within the currently observable field of view. Furthermore, I have reservations regarding the practical utility of this task: the primary purpose of controllable camera motion is to manipulate the camera's movement, not to generate content for novel, previously unseen perspectives. Given that the objective should simply be to ensure geometric consistency within the visible frame, why is it necessary to mandate the generation of content outside the current field of view?

Critical strengths
  1. The paper proposes an intra-policy self-distillation approach, which experimental results show to be quite effective.
  2. The paper constructs a dataset specifically for this task, making a significant contribution to the open-source community.
  3. The experimental results are promising, demonstrating the effectiveness of the proposed method.
Critical weaknesses
  1. I have reservations regarding the paper’s claim that large-scale camera movements lead to issues with generated content appearing outside the field of view. To my knowledge, controllable camera-based video generation methods—such as Wonder—do not exhibit similar problems.
  2. I am also unclear about the task as defined in this paper. What are the practical applications for this task? In what scenarios would one need to specify the location of generated content after the camera movement has already been determined?
  3. 2D bounding boxes offer limited control and fail to convey an object’s orientation. Given the camera-controlled video generation task defined in this paper, why was the 2D bounding box chosen as the control mechanism?
Questions for authors

See Weaknesses.

Reviewer_TA2oOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The manuscript provides a minor modification of Wan2.1-Fun-V1.1-1.3B-Control-Camera (Wan et al., 2025). While previous works provided a text prompt, a frame, and a camera trajectory for guidance, this work uses additionally the scene layout in the last generated frame as guidance. For evaluation, a subset of video clips have been selected from existing datasets and automatically annotated by bounding boxes.

Critical strengths

Since it is minor modification of Wan2.1-Fun-V1.1-1.3B-Control-Camera (Wan et al., 2025) where the layout information is simply added in the same way as the first frame, namely as input to the VAE encoder, it is clear how the additional guidance is added.

The submitted user demo shows how the method could be used. The video is very helpful.

Critical weaknesses

Limited relevance / contribution:

Although the user demo shows how the approach could be used in practice, the problem that is solved in the paper has very little relevance. Providing the layout for the last generated frame is not useful in practice. In the demo, the user draws bounding boxes on a black frame. While the demo shows the bounding box in 3D space, this interface is not useful for users. A more intuitive approach would first generate the video using the baseline and then allow the user to edit the content based on bounding boxes. This is the most straightforward baseline, which is missing in the comparison. Without showing the practical advantage over such an intuitive and practical baseline, the relevance of the proposed method is very low.

The complex training procedure is rather a weakness than a strength. While the paper shows that training using the last layout as guidance performs worse than the student/teacher setup, which requires tracked bounding boxes for all frames F, it indicates that the integration of layout guidance in the same way as the first image might not be the best approach. The results in Table 2, however, show that the gain is not very large (e.g. CLIP 0.23 vs. 0.24). The training procedure is thus not a critical point, but also not a contribution.

The paper does not clarify from the beginning that it is a (minor) extension of Wan2.1-Fun-V1.1-1.3B-Control-Camera (Wan et al., 2025).

Dataset / experimental evaluation:

The dataset itself is not a major contribution. The dataset construction is very specific to the task that is solved. Sequences where the method would not perform better than other methods are discarded. The annotations are derived from the last frame that needs to be predicted - instead of annotating the objects in all frames. The evaluation would be more convincing if datasets, that have been used in previous works, are completely used instead of selecting only sequences that are in favor of the proposed method. It is then still possible to report results for easy and difficult cases depending on the camera motion, but it provides a more fair comparison. Since the proposed approach uses more guidance than other approaches, it should still perform better when there is very little camera motion.

The impact of the loss function weight lambda (10) is not evaluated.

A comparison of the real training time is not provided. It is claimed that "A full cost comparison, including the dense-layout SFT preceding OPSD, is provided in section A.2.1." but it is not included in Appendix A.2.1.

There is no intuition for the results provided. The text in Section 4.2 mainly summarizes the results in the tables, but there are no explanations that provide additional insights.

Presentation needs to be improved:

The dataset description in the paper is not well done and could be moved to the Appendix. The LLM generated Fig. 2 does not add anything. It needs to be better explained how Omega is computed and why all three filter steps (1)-(3) are needed. The method section does not fully introduce the notation. For example, the velocity needs to be introduced. Is it in latent space or image space? Abbreviations like SFT should be introduced. The right hand side of Fig. 4 is unclear.

Questions for authors

What will happen if the layout or text description is inconsistent with the first frame?

How does the dense supervision performs if objects in the last frame are not visible in other frames?

CamC2V: Context-aware Controllable Video Generation (3DV 2026) uses a second image of the scene as guidance. While this is a different setup, the second image is not part of the trajectory. What will happen if the scene layout is not from the last generated frame but from a randomly chosen frame, which is not part of the camera trajectory, as in CamC2V?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.