acceptodds
Under review as a conference paper at ICLR 2027

DisCo: World Models with Discrete Camera Motion Control

Abstract

Controllable video world models target interactive world exploration, where models must faithfully execute explicit action commands while preserving visual quality and temporal coherence. However, most existing approaches rely on continuous camera trajectories as action conditions, which often lead to unreliable action following, especially under complex motion sequences. In this work, we identify action representation entanglement as a key bottleneck in controllable video generation, and show that continuous camera representations lead to high feature similarity across distinct motion patterns, degrading action controllability. Based on this insight, we propose DisCo, a controllable video world model that conditions generation on a compact set of discrete action primitives to improve action separability. We further introduce DisCoBench, a comprehensive benchmark for evaluating the ability of models in short-term, long-horizon, and highly dynamic exploration scenarios. Extensive experiments demonstrate that DisCo achieves significantly more reliable action following while preserving visual quality.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video generation, World model, Camera control

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_JpJVOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

DisCo improves camera-controlled video generation by converting continuous camera trajectories into discrete translation and rotation commands. These actions condition a Wan2.2-based model, which is subsequently distilled into a four-step autoregressive generator. The paper also introduces DisCoBench to evaluate action following, dynamic content generation, and long-horizon consistency.

Critical strengths
  1. The controlled ablation in Table 5 provides useful evidence for the proposed representation. With the backbone, data, and training settings held fixed, discrete actions improve Action Score over both continuous pose and Plücker representations.

  2. The evaluation covers complementary aspects of controllable generation. The appendix strengthens the evidence through human evaluation, multiple evaluation seeds, an external dataset, and continuous pose-error metrics, reducing reliance on the proposed discrete Action Score alone.

Critical weaknesses
  1. The methodological novelty is incremental, and the explanation of the gains is stronger than the evidence supports. Discretization, action embeddings, and the distillation components build on existing approaches. Higher cosine similarity alone does not establish that continuous representations are insufficiently distinguishable or that feature entanglement causes control failures. Table 5 supports the effectiveness of the tested discrete encoding, but not a general disadvantage of continuous representations. The loss of motion-magnitude information and omission of pitch and roll further limit the scope of this conclusion.

  2. No supplementary videos were provided in the materials available for this review. This limits direct assessment of action transitions, motion smoothness, dynamic content, and long-horizon consistency, which are central to the paper’s claims. Static figures and aggregate metrics provide useful evidence but cannot adequately reveal temporal artifacts or visually demonstrate the reported improvements.

Questions for authors

Can the authors strengthen the continuous-representation baseline, for example through improved normalization or an explicit feature-separation objective, to test whether the gains arise from discretization itself or from the particular encoding design?

4 comments

Q1. Methodological novelty beyond existing components.

A1. Thanks for raising this concern. Our contributions are a structured camera-action conditioning design for reliable action following, a controlled study of its representation-level benefits, and DisCoBench for evaluating complex visual exploration. We do not claim quantization, embedding lookup, or distillation themselves as new contributions.

This is non-trivial because assigning discrete labels alone does not specify how translation and rotation semantics should condition video generation or ensure reliable execution when commands change and combine over time.

To address this, we introduce separate learnable translation and rotation embeddings and fuse their projected combination directly with video features (Sec. 3.3), providing explicit conditioning for compositional camera commands. Our representation analysis examines the resulting feature structures (Sec. 4.2), and the controlled ablation in Table 5 establishes the benefit of this design: with the same backbone, data, and training settings, Action Score improves from 0.5245/0.5563 for pose/Plücker encodings to 0.6332.

We further introduce DisCoBench to evaluate command following and transitions, prompted dynamic content, and long-horizon consistency (Sec. 4). This benchmark complements the conditioning design by assessing whether camera commands are executed reliably while maintaining video quality and temporal coherence across these settings.

Q2. Evidence for representation gains and limitations of the action space.

A2. Thank you for highlighting these distinctions. Feature similarity is an empirical characterization of the learned action-conditioning space. In Sec. 4.2, Fig. 4 shows structured similarities among DisCo commands sharing translation or rotation components. Table 4 reports lower feature similarity for DisCo (0.7029 versus 0.8826–0.9913 for the evaluated continuous-action baselines), alongside better action following. These observations provide empirical support for our motivation to make action conditions more distinguishable.

The controlled ablation further establishes the benefit of changing the encoding. With the same backbone, data, architecture, and training settings, Table 5 reports an Action Score of 0.6332 for DisCo, versus 0.5245/0.5563 for pose/Plücker encodings, corresponding to gains of 10.87 and 7.69 percentage points. Our conclusion concerns the effectiveness of this design relative to the tested encodings under matched conditions. We agree that developing stronger continuous representations is a valuable direction; our focus here is a compact action interface for reliable execution of compositional camera commands.

Regarding the action space, we acknowledge that the current interface covers horizontal translation and yaw; it does not specify exact displacement or rotation angles and does not include pitch or roll, as discussed in Appendices B.2–B.3 and E(F in the revision). Appendix B.3 explains the choice of a compact vocabulary to maintain sufficient training samples per category. Table 9 further evaluates finer translation-direction vocabularies: the 27-, 39-, and 48-action variants score 0.6332, 0.5638, and 0.5073, respectively, supporting the compact design for action following under the evaluated conditions.

The discrete framework also offers a path to richer controls. Appendix B.3 discusses adding primitives for different movement speeds or rotation angles, while Appendix B.2 discusses extending the motion categories to pitch and roll. Combining direction commands with discrete magnitude tokens could approximate desired trajectories at a chosen bin resolution. These extensions would require appropriate training data and validation, and remain future work.

Q3. Strengthening the continuous-representation baselines.

A3. Thanks for this suggestion. Our continuous-versus-discrete comparison was specifically designed as a controlled ablation to ensure a fair evaluation. As described in Sec. 4.3 and Appendix B.4, the continuous pose, continuous Plücker, and discrete variants use the same Wan2.2 backbone, training data, training settings, and computational resources, varying only the camera-motion representation. Under these matched conditions, Table 5 reports Action Scores of 0.5245, 0.5563, and 0.6332, respectively. The discrete encoding therefore improves action following by 10.87 and 7.69 percentage points over the two continuous variants under the same training protocol.

We agree that improved normalization or an explicit feature-separation objective could further strengthen continuous encodings. Such experiments would complement the existing fair comparison by testing additional encoding or optimization choices. The current controlled results already support the effectiveness of our discrete design relative to the evaluated continuous representations.

Q4. No supplementary videos were provided in the materials available for this review.

A4. We provide video demos in the newly submitted supplementary material (demo_videos.mp4)

Reviewer_JpJVOpenReview ↗

Official comment by reviewer

Thank you for the clarification. The contribution and the limitations of the action space are now clearer. I will keep my weak-accept recommendation

Thank you for your positive feedback! We sincerely appreciate your time and effort throughout the review process.

Reviewer_tyjZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents DisCo, a video world model that uses discrete camera actions to improve controllability. The authors argue that continuous camera motion representations can make different actions difficult to distinguish, leading to poor action following. By using discrete action embeddings, DisCo aims to improve control while maintaining video quality and long-term consistency. The paper also introduces a benchmark to evaluate action following and video generation, and shows improved performance over existing methods.

Critical strengths
  • Detailed empirical analysis of how action representation affects learned conditioning features and complex-action following provide insight on the performance.
  • The DisCoBench could be a useful evaluation suite for the community to build on.
  • The paper is well-written and easy to follow.
Critical weaknesses
  • It appears that camera quantization has already been introduced by previous work (e.g. DeepVerse, Yume). While the analysis and benchmark of this paper provide some useful insight, the significance of these contributions are questionable.
  • The paper formulates world modeling entirely in terms of camera-motion control. This is not the standard setup of a world model where the action includes not just camera motion but also agent actions (e.g. the movement of a robot). Such a distinction should be made clear.
  • Section 4.2 reports that continuous action features have higher average cosine similarity, while discrete features have lower similarity. However, the authors do not establish that high similarity itself causes poor action following. This results in insufficient evidence that feature entanglement causes the controllability problem.
  • While the proposed discrete action representation improves directional action-following performance, it is unclear how the model can support arbitrary continuous camera trajectories during autoregressive rollout. In particular, different translation magnitudes and rotation angles are mapped to identical discrete action labels, introducing an inherent loss of control information.
Questions for authors

See weaknesses

4 comments

Q1. Novelty beyond camera quantization in DeepVerse and Yume.

A1. Thanks for raising this. Our contribution concerns how camera actions are represented and used, beyond quantization itself. Sec. 3.2 already credits DeepVerse for the quantization procedure. Below, we explain how our action conditioning differs from these works and summarize the supporting experimental evidence.

Conditioning design. DeepVerse derives textual camera-motion descriptions from pose changes (its Sec. 2.3), while Yume maps quantized motions to text conditions (its Sec. 5.4). DisCo instead uses separate learnable translation and rotation embeddings, directly fused with video features (Sec. 3.3, Eq. (2)). This provides dedicated action conditioning for changing and compositional commands.

Controlled evidence and evaluation. Sec. 4.3 isolates representation choice under matched backbone, data, and training settings: Table 5 reports Action Score gains of 10.87 and 7.69 percentage points over pose and Plücker encodings. DisCoBench complements this study with evaluations of complex commands, dynamic content, and long-horizon consistency (Sec. 4).

Q2. Scope of the term world model.

A2. Thank you for pointing this out. We agree that general-purpose world models may involve a broader range of actions, including agent interactions and object manipulations, beyond camera motion alone.

This work focuses on camera-controlled video generation for visual exploration. Following prior work on interactive video world models, such as DeepVerse and Matrix-Game2, we use the term in the context of camera-conditioned generation of visual observations during exploration. We further make this scope explicit in the Appendix B.5 of the revised version.

Q3. Does feature similarity establish the cause of control failures?

A3. Thanks for highlighting this distinction. Feature similarity is an empirical observation of how different camera actions are represented in the learned conditioning space. We examine this feature structure together with action-following performance and use a controlled ablation to evaluate the benefit of the representation choice.

In Sec. 4.2, Fig. 4 shows that DisCo’s action features exhibit structured similarities among commands sharing translation or rotation components. Table 4 reports a feature similarity score of 0.7029 for DisCo, compared with 0.8826–0.9913 for the evaluated continuous-action baselines, alongside a higher Action Score of 0.6332 versus 0.2736–0.5731. These observations are consistent with our motivation to provide more distinguishable action conditioning for reliable command execution.

The controlled ablation in Sec. 4.3 further establishes the benefit of changing the encoding. With the backbone, training data, architecture, and training settings held fixed, Table 5 shows that replacing continuous pose and Plücker encodings with our discrete encoding reduces feature similarity from 0.9377/0.9201 to 0.7029 and improves Action Score from 0.5245/0.5563 to 0.6332. Together, these analyses provide empirical support for our representation design and its benefit for action following.

Q4. Arbitrary continuous trajectories and motion-magnitude control.

A4. Thank you for raising this concern. Our focus in this work is reliable execution of compositional camera commands for visual exploration. Sec. 3.2 represents these commands as sequences of 9 translation and 3 yaw primitives, supporting their composition during autoregressive rollout. We acknowledge that this compact interface does not retain exact translation magnitudes or rotation angles. As discussed in Appendix B.3, this is a deliberate trade-off to maintain sufficient training samples for each action category. Our results in Table 5 and Table 9 support the benefit of this choice for action following, favoring the compact vocabulary for reliable command execution under the evaluated data and training conditions.

Appendix B.3 already discusses adding discrete primitives for different movement speeds or rotation angles. One possible implementation is to augment the existing direction commands with separate displacement/speed and rotation-angle tokens. A desired continuous trajectory could then be approximated as a sequence of commands specifying both direction and magnitude, with precision determined by the discretization resolution. This offers a way to extend trajectory control within the discrete framework.

Dear Reviewer,

As the discussion phase is approaching its end, we hope that the above clarifications have addressed your concerns. If you find our responses satisfactory, we would greatly appreciate your consideration of updating the score to reflect the additional results and discussion. We remain happy to clarify any remaining points during the discussion phase.

Thank you again for your time and consideration.

I thank the authors for their rebuttal effort. The rebuttal partially addresses my concerns (e.g. regarding the distinction from prior work, the scope of the world model, and the trade-off between discrete and continuous control). However, my main concerns about the limited methodological novelty remain. As such, I will maintain my original score.

Thank you for your comment. We would like to further emphasize that, unlike DeepVerse and Yume, which convert discrete actions into textual conditions, DisCo employs separate learnable translation and rotation embeddings that are directly fused with video features (Sec. 3.3, Eq. (2)). This enables dedicated action conditioning for flexible and compositional commands.

Beyond this architectural distinction, we conduct controlled studies that isolate the effect of action representations under identical backbones, datasets, and training settings, providing representation-level insights supported by controlled evidence for future research in controllable video generation. We also introduce DisCoBench, a dedicated benchmark evaluating complex commands, dynamic scenes, and long-horizon consistency, which requires substantial effort in benchmark design and curation.

We appreciate that the reviewer has acknowledged our distinction from prior work, and we hope these architectural contributions, representation-level insights supported by controlled evidence, and benchmarking efforts further clarify the novelty and significance of our work.

Reviewer_SvruOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This project proposes a method to train interactive video generation model (or world model) with discrete actions, instead of continuous action embedding which is usually adopted. Through these discrete action representation, the action conditions are separated more clearly and can be easier for model to learn. The model shows greater action-following capabilities without leading to general video generation quality degradation. This work also introduce a new benchmark for evaluating the action following capabilities of video models.

Critical strengths
  1. clear visualization of feature-level similarity landscape, which serves as an interesting and strong motivation of this paper.
  2. impressive quantitive and qualitative results, demonstrating the significance of discrete labels in the model's action following capabilities.
  3. a comprehensive work, including self-build benchmark to evaluated the corresponding action-following in video world models.
Critical weaknesses
  1. The discrete nature of these action representations limits the potential of smooth and precise interactivity in video world model.
  2. Discrete actions used for training and for evaluation are the same, while other models are trained on general continuous actions, which maybe unfair for baselines.
  3. potentially down-streaming applications are limited to game scenarios and maybe cannot be extended to real-world scenes.
Questions for authors
  1. Can authors demonstrate the necessity of discrete action labels? I means can we find a continuous representation for action, which has clear inter-action similarity, instead of using simple discrete actions.
  2. Can the discrete actions be interpolated or extrapolate? If not, can we train a model that has this feature, which can naturally bridge the gap between discrete action and continuous action?
  3. Besides the self-build benchmark, the authors should evaluate the model versus baselines on third-party benchmarks like WBench for fair comparison.
5 comments

Q1. Smoothness and precision with discrete actions.

A1. Thank you for your comment. We would like to address the concerns regarding motion smoothness and control precision separately.

Motion smoothness. Discrete action representations does not necessarily imply discontinuous or less smooth generated motions, since the conditions describe translation and rotation primitives over short temporal segments and serve as conditioning features for the video generator, rather than prescribing discrete jumps between output camera poses. We also measure the temporal smoothness of generated videos in Table 1. The results show our method achieves 0.9842 Motion Smoothness score, versus 0.9824 for Matrix-Game2 and 0.9764 for WanX-Cam. The human evaluation in Appendix D.3, Table 10, also favors DisCo over these two baselines for motion smoothness (73.4% and 69.0% preference, respectively).

Control precision. We acknowledge that, compared with continuous camera trajectories, discretization reduces fine-grained control expressiveness, including the ability to specify exact translation magnitudes and rotation angles, as discussed in Appendices B.3 and E. However, the compact representation yields more reliable action following in our controlled experiments. With the same backbone, data, and training settings, Table 5 reports an Action Score of 0.6332 for DisCo, outperforming continuous pose and Plücker encodings by 10.87 and 7.69 percentage points, respectively. Furthermore, Appendix D.2, Table 9, shows that finer 39-/48-action vocabularies (30°/24° translation bins) score 0.5638/0.5073, below 0.6332 for the 27-action vocabulary (45°) under the same evaluation protocol. These results support our choice to trade some representational precision for stronger action-following performance.

Q2. Fair comparison between continuous and discrete controls.

A2. Thanks for raising this concern. We address fairness through a matched-training representation ablation and continuous pose-error evaluation independent of our discretization rule.

Sec. 4.3 and Appendix B.4 compare pose, Plücker, and discrete encodings with the same Wan2.2 backbone, training data, training settings, and computational resources. Changing only the camera-motion representation gives Action Scores of 0.5245, 0.5563, and 0.6332, respectively (Table 5), isolating this choice from cross-model training differences.

Appendix D.6, Table 13, additionally reports continuous errors on 100 real-world Sekai videos: DisCo achieves RPE-T/RPE-R of 0.054/0.326, versus 0.065/0.394 for WanX-Cam and 0.092/0.544 for Matrix-Game2. Its advantage therefore also appears under metrics independent of the discrete Action Score. Table 1 remains a system-level comparison under each method’s settings, as clarified in Appendix B.4.

Q3. Generalization beyond game scenes.

A3. Thanks! The paper already evaluates DisCo on 100 unseen real-world Sekai videos (Appendix D.4). Table 11 reports Action Score/FVD of 0.602/333.93 for DisCo, compared with 0.567/369.10 for Matrix-Game2.

Appendix D.6 further reports continuous pose errors on the same set. These results provide direct evidence beyond game-scene evaluation and support generalization to the tested real-world exploration scenes.

Q4. Why use discrete labels rather than a better continuous representation?

A4. Thank you for this suggestion. We agree that exploring continuous representations with more distinguishable action features is a promising research direction. In this work, however, we focus on improving action-following reliability through a compact and semantically explicit camera-control interface. We represent translation and rotation primitives using separate learned embeddings that directly condition the video generator (Secs. 3.2–3.3), allowing camera commands to be expressed through explicit motion categories.

Our results also support this design choice. With the same backbone, training data, and training settings, Table 5 reports an Action Score of 0.6332 for DisCo, compared with 0.5245 for continuous pose and 0.5563 for Plücker encodings, corresponding to gains of 10.87 and 7.69 percentage points, respectively (Sec. 4.3). These results demonstrate that the compact discrete representation improves action following over the tested continuous encodings under matched training conditions, supporting its effectiveness for the camera-control task studied here.

Q5. Interpolation and extrapolation between discrete actions.

A5: Thank you for your comment. We have explored interpolation and extrapolation between discrete actions but have not identified a reliable approach that consistently produces the desired intermediate or extrapolated motion directions. We acknowledge this limitation and consider extending the discrete action space to support more flexible and fine-grained motion control an interesting direction for future work.

Q6. Third-party evaluation on WBench.

A6. Thanks for this suggestion. We additionally evaluated DisCo and the baselines on WBench using NavScore to measure action following. DisCo achieves the highest score among the three compared models, providing additional evidence on a third-party benchmark.

ModelNavScore
WanX-Cam73.37
Matrix-Game280.88
Ours82.51

Dear Reviewer,

As the discussion phase is approaching its end, we hope that the above clarifications and additional experiments have addressed your concerns. If you find our responses satisfactory, we would greatly appreciate your consideration of updating the score to reflect the additional results and discussion. We remain happy to clarify any remaining points during the discussion phase.

Thank you again for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.