acceptodds
Withdrawn from ICLR 2027

Beyond Visual Fidelity: Comparing Categorical and Continuous World Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Visual world models can now generate compelling interactive worlds, but visual realism does not guarantee faithful dynamics. To understand how modeling choices affect world model behavior, we present a systematic comparison that disentangles prediction representation from generation procedure, choices often bundled in approaches such as autoregressive token models and whole-frame flow matching. In controlled 2D and 3D environments, we evaluate dynamics validity, calibration, and visual quality. Failures such as disappearing game characters echo those observed in more complex environments, but here their impact can be measured against known dynamics. With lossless visual encoding in 2D, discrete- token models achieve higher rollout validity at lower training compute, particularly for stochastic entities. However, high validity can coexist with poor calibration: generating permissible trajectories does not imply reproducing their probabilities. In 3D, visual degradation from lossy tokenization does not consistently translate into worse long-horizon metrics. These results show how representation and decoding choices affect validity and calibration differently, demonstrating the importance of evaluating world models beyond visual fidelity.

Primary area:
generative models
Keywords:
world models, discrete representations, continuous representations, flow matching, autoregressive modelling

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_qX7YOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents a controlled comparison of categorical and continuous visual world models, crossed with next-token vs. iterative frame generation. It evaluates four architectures in a lossless Pixel Pac-Man environment and a simple 3D maze, focusing on transition and rollout validity, stochastic calibration, visual fidelity, and training compute.

Critical strengths
  1. Strong controlled-study design. The 2x2 design is more diagnostic than comparing unrelated end-to-end world models. By crossing representation type with generation procedure, the paper attempts to isolate which modeling choices are responsible for differences in dynamics quality.

  2. Known dynamics enable meaningful evaluation. In Pixel Pac-Man, the simulator rules are known, so the authors can distinguish legitimate stochastic alternatives from actual dynamics violations rather than penalizing disagreement with a single sampled reference trajectory. This makes transition and rollout validity much more informative than standard perceptual similarity metrics.

  3. The validity against calibration distinction is useful. A particularly strong result is that high transition validity does not imply calibrated stochastic predictions. The paper shows that models can generate mostly permissible futures while still assigning incorrect probabilities to those futures, motivating separate evaluation of validity and calibration.

Critical weaknesses
  1. The environments may be too toy to support broad representation-level conclusions. The controlled environments make exact validity analysis possible, which is a strength, but they are also highly simplified. Pixel Pac-Man has a small discrete state space with hand specified dynamics, while the 3D maze uses solid colors, no textures or shading, and simple non-interacting bouncing spheres. These settings are far from the visually complex, contact-rich, partially observed environments that motivate modern world models. As a result, it is unclear whether the observed categorical advantage reflects a general property of world-model representations or an advantage specific to low-entropy synthetic environments whose structure is especially compatible with discrete tokenization. Evaluation on at least one substantially richer environment would be important for establishing external validity.

  2. There are potential confounds in claimed 2x2 disentanglement. The paper frames the experiment as crossing representation with generation procedure, but the next token and iterative variants also use different transformer structures, namely decoder-only causal blocks versus DiT-style block-causal architectures. Their objectives and decoding procedures also differ substantially. Thus, the generation procedure factor bundles several architectural and training differences, which might posit confounds. The categorical vs. continuous comparisons are relatively clean, but broader interpretations across next token vs. iterative generation should be more cautious.

  3. Related work could be improved. The beyond visual fidelity motivation overlaps substantially with recent work on world-model evaluation beyond perceptual quality. Examples include but are not limited to WorldModelBench (https://arxiv.org/abs/2502.20694), WorldScore (https://arxiv.org/abs/2504.00983), and World-in-World (https://arxiv.org/abs/2510.18135). These works do not replace the contribution here, since the controlled representation study is distinct, but they weaken the framing that large-scale evaluation is primarily visual-quality-driven. The novelty could be better positioned as isolating how representation and generation choices affect validity and calibration under controlled dynamics, rather than establishing the need to move beyond visual fidelity itself.

Questions for authors

I am overall open to increase my score following the discussion with the authors. Below are my questions related to the concerns I raise above:

  • Do the categorical advantages persist in an environment whose native state is genuinely continuous rather than discretely structured?
  • Can the authors provide multi-seed 3D results and simulator-state validity metrics analogous to Pac-Man?
  • How should readers separate the effect of generation procedure from the accompanying causal-transformer vs. DiT architectural change?
Reviewer_kVp9OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the differences between categorical token-based models and continuous generative models for visual world modeling, arguing that world model performance should not be evaluated solely based on visual quality. The authors introduce a unified 2×2 comparison framework and analyze how different representations and generation strategies affect dynamics modeling in the Pixel Pac-Man and 3D Maze environments. Experiments show that categorical token models generally achieve better long-horizon rollout stability and stochastic dynamics modeling, while the visual advantages of continuous models do not necessarily translate into stronger world understanding. The work highlights that world models should be evaluated beyond visual fidelity by considering dynamics validity, long-term consistency, and probability calibration.

Critical strengths
  1. The research problem is important and practically relevant This paper investigates a key question in visual world models: whether visual fidelity truly reflects an understanding of real-world dynamics. It points out that existing approaches often focus excessively on generation quality while overlooking long-term consistency and adherence to environmental rules. This study has important implications for robotics, embodied AI, and interactive AI systems.

  2. It proposes an evaluation framework beyond visual quality This paper introduces a more comprehensive evaluation framework for world models by considering metrics such as transition validity, rollout validity, and calibration, rather than relying only on visual quality. This provides a new perspective for analyzing and comparing different world model architectures.

Critical weaknesses
  1. The experimental environments are relatively simple, limiting the generalizability of the conclusions This paper mainly evaluates models in controlled environments such as Pixel Pac-Man and 3D Maze. Although these environments enable precise analysis of model dynamics, they still differ significantly from real-world tasks. Whether the conclusions remain valid for complex scenarios such as robotic manipulation, autonomous driving, or large-scale video world models requires further investigation.

  2. The model and experimental scales are limited The experiments are conducted using relatively small-scale Transformer models and limited training compute budgets, which differ substantially from current large-scale world models. Therefore, it remains unclear whether the observed advantages of different modeling approaches will persist or change as model scale increases.

Questions for authors
  1. Fairness of the comparison between categorical and continuous models This paper provides a unified framework to compare categorical and continuous world models. However, the two types of models still differ in several aspects, including model architecture, training objectives, and inference procedures. Therefore, it remains unclear whether the observed performance differences are mainly caused by the representation choice itself. The authors could further report model sizes, training/inference costs, and provide more analysis on how different design choices contribute to the final performance.

  2. Understanding the source of categorical models’ advantages The paper shows that categorical models achieve better long-horizon rollout performance and stochastic dynamics modeling. However, it is still unclear whether this advantage mainly comes from the discrete representation itself or from the constraints introduced by discrete tokenization. For example, in Pixel Pac-Man, categorical models directly predict valid state tokens, while continuous models need to model the dynamics in a continuous space and map the outputs back to discrete states. Further analysis of how different representations affect performance would help better understand this observation.

  3. Generalization of the experimental findings The paper evaluates models mainly on controlled environments such as Pixel Pac-Man and 3D Maze. These environments are useful for analyzing world model dynamics, but they are still different from real-world applications. In addition, the evaluated models are relatively small in scale, and it remains unclear whether the observed advantages of categorical models will hold for more complex environments and large-scale world models. Additional experiments or discussions on scaling effects would help strengthen the conclusions.

Reviewer_hx1uOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents a controlled comparison of categorical and continuous visual world models, crossing output representation with patch-level autoregressive versus iterative whole-frame generation. The resulting four-model design is evaluated in a rule-based Pixel Pac-Man environment and a partially observed 3D maze. The study distinguishes transition and rollout validity from distributional calibration and visual fidelity. Experiments show that categorical models maintain valid Pac-Man trajectories at lower training compute, particularly with stochastic entities, while model rankings differ under calibration. In the 3D maze, lossy tokenization reduces next-frame visual accuracy without consistently worsening long-horizon semantic accuracy or FVD.

Critical strengths
  1. The paper addresses a meaningful evaluation problem: visually plausible predictions need not preserve environmental rules or reproduce the correct distribution of future outcomes. The distinction between validity and calibration is clearly motivated and empirically illustrated.
  2. The authors provide detailed analyses, including compute-dependent comparisons, component-level failures, varying entity counts and policies, and inference-parameter studies. These analyses help characterize the observed behavior and provide strong empirical support for the findings within the evaluated settings.
Critical weaknesses
  1. The representation-level conclusions are not sufficiently separated from the particular encoding, training, and capacity choices. Although the four-model design controls for the within-frame generation procedure, representation remains coupled with different training objectives, conditioning mechanisms, and output heads. In particular, the continuous models operate directly on RGB using small backbones, without sufficient investigation of alternative encodings or larger capacities. Consequently, it remains unclear whether the observed differences arise primarily from categorical versus continuous representations or from the specific implementations and capacity allocations.
  2. The findings have limited demonstrated applicability to more complex world-modeling settings. Realistic visual environments typically require learned compression, where discrete and continuous autoencoders can differ substantially in reconstruction quality and information retention. The controlled experiments provide useful observations, but do not yet establish whether the same conclusions hold for world models in more complex environments.
Questions for authors
  1. How robust are the results to backbone capacity? Could the authors report capacity sweeps for all four methods, together with sufficiently trained, near-converged results, to distinguish differences in training efficiency from differences in achievable performance?
  2. The 3D evaluation currently relies on visual and semantic agreement metrics. Could the authors introduce dynamics-validity measures analogous to those used in 2D, such as object persistence, motion consistency, and collision correctness?
  3. How would the continuous models perform when predicting lossy VAE latents rather than RGB directly? A comparison with the VQ-VAE setting would help assess the relevance of the findings to practical world models.
  4. Do the reported training FLOPs include tokenizer or VQ-VAE training? Please also report inference FLOPs and latency to better distinguish training efficiency from generation efficiency.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.