acceptodds
Withdrawn from ICLR 2027

Holo-World: Unified Camera, Object and Weather Control for Video World Model

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

16 views

Abstract

Video world models are moving toward preserving an observed world under controllable camera and object motion while allowing its environmental state to change. Yet these controls remain isolated, and weather generation typically relies on a source video or reconstructed scene that already specifies future structure. We study a first-frame-anchored source-to-state setting, where the model starts from a single image and follows explicit camera and object controls and an optional weather instruction, then generates a video that either preserves the source world or transfers it to a target weather state. To address these challenges, we first build HoloStateData, a state video dataset that turns diverse videos into unified control samples for camera, object, and weather supervision. Second, we introduce Holo-World, a unified controllable video world model that jointly controls the scene from a single image. Its Unified Scene Adapter factorizes world preservation and weather transfer into distinct parameter subspaces, using rendered background, geometry buffers, and object controls to maintain controlled scene structure while modeling weather-dependent appearance and particle effects. Additionally, Scene-Weather Decomposed CFG guides scene and weather residuals separately, strengthening target weather effects without over-amplifying the full condition. Quantitative and qualitative experiments demonstrate that Holo-World maintains precise camera and object controls with consistent scene structure while transferring scenes into diverse target weather states, outperforming video-to-video weather editing baselines on weather-state generation. Our project page is available at https://anonymous.4open.science/w/Holo-Anonymous-Page/, and an offline page with video demonstrations is included in the supplementary material.

Primary area:
generative models
Keywords:
Video World Model, Video Diffusion, Camera Generation, Weather Transfer

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_EqxJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies video generation from a single input image with joint control over camera motion, object motion, and weather. Built on a frozen video generation backbone, the method uses a World Adapter to incorporate rendered scene geometry and object bounding-box trajectories, and a State Adapter to introduce weather conditions. Scene–Weather CFG separately weights scene and weather guidance. Training combines real videos, paired simulated weather data, and weather pairs produced through video editing. The intended contribution is to change weather while preserving the underlying scene and prescribed motion.

Critical strengths

The problem is meaningful and practically relevant. Jointly controlling camera motion, object motion, and weather addresses a useful capability in controllable video generation. Preserving scene structure and object consistency under substantial appearance changes is a worthwhile research objective.

The paper presents a coherent system and data construction pipeline. The integration of geometric conditions, object trajectories, and weather prompts is clearly motivated. The reported results provide initial evidence of feasibility, while the ablations offer some support for the geometric inputs and guidance strategy.

Critical weaknesses

The comparisons do not sufficiently establish the benefit of the proposed joint architecture over representative existing systems and straightforward compositions. Recent systems such as WorldPlay, LingBot-World V2, Matrix-Game 3.0, and Cosmos 3 are relevant to camera-conditioned or interactive world generation, while Wan-Move is relevant to multi-object motion control. WorldPlay and Wan-Move are cited but not evaluated. More importantly, two natural alternatives are missing: editing the input image into the target weather before controllable video generation, and applying weather editing after controllable video generation. Both can operate from the same initial image and user instructions without additional observed future frames. Without these comparisons, it remains unclear how much the dual-adapter architecture and joint training improve quality, control accuracy, or computational cost over compositions of existing capabilities.

The evaluation provides limited evidence for preserving joint control under weather changes, particularly in multi-object settings. Weather quality and structural preservation are largely assessed through separate metrics and subsets. Appendix Table 7 does report control errors on the Weather subset, but lacks corresponding baseline results and matched guidance-strength comparisons. The background-preservation metrics in Table 5 are evaluated on the Real subset and therefore do not establish whether stronger weather guidance compromises structure or motion control on weather-conditioned samples. Multi-object robustness also remains insufficiently characterized. The paper does not adequately report object-count distributions or performance under same-category crossings, occlusion, and reappearance. Bounding boxes specify spatial constraints, but stable identity-to-trajectory binding requires further evidence. ObjMC, based on trajectory-center distances and Hungarian matching, does not by itself establish appearance-identity consistency.

Finally, the background evaluation mask described in the appendix depends on both the reference and generated outputs. This appears to allow some erroneous generated regions to be excluded from evaluation. The implementation should be clarified, and the conclusions checked using a valid region determined solely from reference information.

Questions for authors

Baseline selection: What criteria guided the selection of baselines? Could the authors provide representative comparisons with recent camera/world-control and object-motion methods, such as those discussed above, on their shared capabilities? Please clarify differences in conditioning information, model budgets, and practical availability.

Compositional alternatives: How does the proposed method compare with (a) first-frame weather editing followed by controllable generation and (b) controllable generation followed by video weather editing? Comparisons should jointly assess weather quality, scene and object preservation, control accuracy, and inference cost. For the first alternative, any downstream RGB conditions should be made consistent with the edited appearance.

Weather–control trade-off and metric validity: On the same Weather samples, how do camera accuracy, object accuracy, and scene preservation change when weather control is disabled or its guidance strength is varied? Does the background evaluation mask depend on generated pixels in the implementation, how are empty masks handled, and do the conclusions hold with a reference-only mask?

Multi-object control: How are object identities encoded and associated with trajectories, and how are overlapping boxes represented? Could the authors report object-count statistics and results stratified by object count, together with targeted evaluation of crossings, occlusion, and reappearance? How does the evaluation detect identity swaps, trajectory reassignment, or disappearing objects?

Synthetic pair quality: Which editing models and versions were used to construct the weather pairs, and in what proportions? Please clarify the filtering procedure, rejection rate, and residual-error assessment. Is there a quality audit or data ablation that measures the effect of geometric changes, object distortions, and temporal artifacts in the edited targets?

Reviewer_SaYaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents Holo-World, a video generation framework that jointly supports camera, object motion, and weather control from a single input image. The authors construct HoloStateData, a dataset combining real videos, simulated weather pairs, and video-to-video weather transfer samples with structured geometric and motion annotations. The proposed method builds on a frozen Wan2.1 backbone and introduces two separate adapters: a World Adapter for scene preservation and geometric control, and a State Adapter for weather transformation. A decomposed classifier-free guidance strategy is further proposed to control scene and weather conditions independently during inference. Experiments show improvements in camera and object control accuracy, as well as weather generation quality.

Critical strengths
  1. The paper addresses a practical problem by combining camera motion, object dynamics, and weather changes within a single video generation framework. Unlike conventional video-to-video weather editing, the proposed setting requires generating future frames from an initial image under explicit controls. This provides a useful extension of existing controllable video generation methods.
  2. Holo-World achieves competitive camera and object control accuracy compared with existing methods, while also supporting weather generation. The weather transfer results are encouraging, particularly the improvements over existing video editing baselines despite using only a single source frame. The experiments suggest that the proposed framework can integrate multiple control capabilities without substantially compromising generation quality.
Critical weaknesses
  1. The main technical contributions consist of two separate control adapters and a decomposed CFG strategy. Both are relatively straightforward extensions of existing conditional video generation techniques. More importantly, the paper does not convincingly establish why separate adapters are necessary. The current ablations do not include a parameter-matched single-adapter baseline that jointly handles all conditions. In Table 4, the improvement from introducing UniSA is relatively limited, while much of the final gain comes from SW-CFG. This makes it difficult to assess the actual contribution of the proposed architecture.
  2. The paper emphasizes joint camera, object, and weather control, but the evaluations largely assess these capabilities separately. Camera and object control are mainly compared on the Real subset, while the primary weather metrics evaluate weather appearance without considering scene consistency or motion accuracy. Although the appendix reports control errors on the Weather subset, there is no corresponding baseline comparison or systematic analysis of how weather changes affect geometric and motion consistency. The paper also reports that stronger weather guidance can damage scene structure, but provides limited quantitative evidence for this trade-off. Consequently, the results do not fully demonstrate that the proposed method effectively resolves interference between different controls.
Questions for authors
  1. Is the separation into World and State Adapters necessary? Could the authors compare UniSA against a parameter-matched single adapter trained with the same conditions and SW-CFG? This would help clarify whether the architectural separation provides benefits beyond the guidance strategy.
  2. How does weather control affect camera and object control accuracy? Could the authors report camera, object, and geometry consistency metrics on the Weather subset under different weather guidance strengths? This would directly evaluate the claimed benefit of separating world preservation and weather generation.
  3. How are scene prompts obtained during evaluation? The dataset construction uses complete video clips to generate scene descriptions, including temporal evolution and object activities. Are these descriptions also used during evaluation, or are scene prompts generated solely from the first frame? This distinction is important for understanding the actual information available to the model in the claimed single-image generation setting.
Reviewer_JowiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents Holo-World, a video generation model that combines camera, object, and weather control from an input image and structured controls. It introduces HoloStateData, which combines real videos, simulated weather pairs, and generated weather-editing pairs. The model uses separate adapters for scene preservation and weather transfer, together with separate guidance terms during sampling. Experiments evaluate video quality and control accuracy on a Real subset, and weather generation on a Weather subset.

Critical strengths

The paper is generally clear. The data pipeline and model design are easy to follow.

Joint camera, object, and weather control is a meaningful extension of controllable video generation. Supporting these controls in one model could simplify workflows that currently require separate generation and editing stages.

The pipeline separates scene and weather descriptions and constructs controls from the source side. Combining real, simulated, and generated pairs provides a practical way to train both scene preservation and weather transfer.

Critical weaknesses

The methodological novelty is relatively limited. The two-branch adapter in Section 4.2 closely resembles ControlNet-style residual conditioning, while Section 4.3 applies separate CFG scales to scene and weather prediction differences. These are reasonable design choices, but the paper does not clearly explain what is technically new beyond combining existing ideas for camera, object, and weather control. The data pipeline is thoughtful, but the novelty of the model and sampling method remains limited.

In addition, comparisons are missing to establish the value of the proposed design. The Weather evaluation compares against V2V editors with different input conditions. Despite repeatedly emphasizing that Holo-World is not a cascaded editor, the paper does not compare against a direct cascade: generate a video with its World Adapter or other baselines, then apply Wan2.7-Edit or V2V. Without further comparisons, it is difficult to tell whether the proposed architecture offers a meaningful benefit over simpler alternatives.

Questions for authors

[Q1] Line 192: "a combination of our weather-transfer model and proprietary video editing models"

Which models are used, and how much does Holo-World depend on their outputs? Since these models can already perform weather editing, what benefit does Holo-World offer over first generating a controlled video and then applying one of these editors?

[Q2] Source depth and normal maps may not fully match the target weather appearance. E.g. rain and fog can obscure surfaces, accumulated snow can change their visible geometry. How does the model handle conflict between fixed source geometry and the target weather?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.