acceptodds
Under review as a conference paper at ICLR 2027

Can Video World Models Track Unobserved World States?

Abstract

Video world models are increasingly used as simulators, but visual fidelity alone does not show that a model maintains the hidden state of the world. We examine this difference with an action-conditioned video Shell Game, a visual analogue of state tracking that separates visual rendering from compositing the unobserved world state. Trained on 5-swap chains, standard backbones (e.g., bidirectional and autoregressive Transformers, Mamba, and linear attention) render plausible videos and predict the correct ball location up to 5 swaps. However, they fail to learn the rule and generalize to longer swap chains, even with more denoising steps. As the pixel-based diffusion loss does not force the generated frames to hold the unseen ball position, output tokens cannot carry it, and the state has to live within the architecture. In a causal Transformer, this implicit state is an append-only KV cache, which is written once and never revised, so the model must re-compose the swaps at every chunk. Tracking this way requires depth to grow with sequence length, which no fixed-depth Transformer provides. We study what enables learning the rule, and find that length generalization requires a revisable state carried across chunks and an update expressive enough to apply a swap. Linear attention can achieve this by allowing negative transition eigenvalues, and autoregressive Transformers can do so with nonlinear TTT fast weights (e.g., SwiGLU) whose online updates change the feature map used to read their state. We further examine Memory Maze and Block World, where the state is not fixed by the input action stream alone and must be corrected from observations or keeps changing out of view, and discuss the implications for building stateful video world models.

Primary area:
generative models
Keywords:
Video world model, Video diffusion, Autoregressive Video Diffusion, Model expressivity, State tracking

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_1oV2OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

Video world models generate fidelity scenes but may not maintain the hidden state of the world. This paper uses an action-conditioned video shell game as the testbed. Experiments show that standard backbones could render plausible videos but fail to generalize to longer swap chains. The authors point out that the pixel-based diffusion loss does not force the generated frames to hold the unseen ball position. The authors also note that in causal transformer, the implicit state is an append-only KV cache, for which the model must re-compute the swaps at every chunk. Then, the paper studies what enables learning the rule. The finding is that length generalization requires a revisable state carried across chunks and an update expressive enough to apply a swap.

Critical strengths
  • The paper studies a critial but often omitted problem: the state transition in a scene. It provides insights on building more stateful world models. Beyond controllability or fidelity, enabling state transition tracking is also important for world models.

  • The experiment uses a clear and meaningful diagnostic task, Shell Game, which separate video generation and state tracing, making it possible to study the hidden-states in different model types. The accompanying symbolic experiments help distinguish difficulties in composing state transitions from those introduced by visual learning. A shared video backbone and approximately matched parameter budgets further improve the interpretability of the architectural comparisons.

  • The evaluation spans Transformers, recurrent networks, state-space models, linear attention, and nonlinear fast weights. Beyond comparing overall performance, the paper examines sequence length, attention windows, denoising steps, transition eigenvalues, fast-weight heads, and which parameters receive online updates. These complementary experiments address several plausible explanations for success and failure.

Critical weaknesses
  • The Shell Game isolates action-driven permutation tracking effectively, but provides limited evidence about state estimation that requires integrating observations and correcting prior beliefs. Real world tasks are more complex.

  • The additional benchmarks provide limited direct validation of hidden-state tracking. Their evaluation primarily measures visual reconstruction quality, leaving it unclear whether improvements reflect more accurate state maintenance or better retrieval and rendering.

Questions for authors

None

Reviewer_NY7XOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates whether action-conditioned video models can track world states that are temporarily unobserved. It introduces a visual Shell Game in which five identical cups undergo swaps while a ball remains hidden, separating accurate rendering from correct hidden-state tracking. Models are trained on five-swap episodes and evaluated on longer chains.

Critical strengths
  1. The benchmark cleanly distinguishes visual fidelity from hidden-state correctness. Identical cups and an occluded ball make plausible intermediate frames insufficient to determine success; the final reveal provides a directly interpretable state-accuracy measure. The length and denoising-budget sweeps demonstrate that good reconstruction metrics can coexist with incorrect hidden state. This is a useful diagnostic contribution for evaluating video models as simulators, beyond conventional video-quality comparisons.

  2. Several controlled interventions provide informative evidence about state-update mechanisms. Positive- and negative-eigenvalue variants retain the same dimensions and parameter counts, and the latter consistently improve extrapolation across multiple linear-attention families; GDN-neg reaches approximately 0.95 accuracy at 30 swaps. Table 1 further shows that updating only the LaCT readout fails to extrapolate, whereas updating its nonlinear feature map with the readout frozen preserves most of the benefit. These comparisons make the paper more informative than a backbone ranking. The contribution is primarily an empirical investigation in video generation, building on established state-tracking theory and negative-eigenvalue mechanisms.

Critical weaknesses
  1. The Transformer experiments do not fully isolate expressivity limitations from context truncation and positional extrapolation. Footnote 4 states that the causal full-attention model is evaluated with a seven-chunk history window beyond the training horizon because unrestricted-context generation suffers from unseen RoPE offsets. Consequently, longer evaluations discard the initial reveal and earlier swaps. Appendix H explicitly acknowledges this truncation-induced distribution shift. These results establish failure of the tested inference configuration, but do not cleanly attribute finite-horizon failure to fixed-depth permutation composition alone. The cited asymptotic complexity results motivate the investigation, yet do not establish that failure at 10–30 swaps follows from an expressivity barrier. Similarly, unsuccessful linear decoding from fixed-bank readouts is suggestive rather than proof that hidden-state information is absent. The broad claims should be restricted to the tested training and inference setup unless stronger controls are provided.

  2. Some explanations of why particular memory mechanisms work remain confounded. In Table A2, increasing LaCT heads from one to eight also increases backbone depth from eight to twelve layers. Thus, the head sweep does not independently establish that factorizing the update into more heads causes better tracking despite smaller state size. Separately, the scratchpad comparison attributes failure to exploding memory norms, while LaCT explicitly normalizes its fast weights after updates; a normalized or gated scratchpad control is missing. These limitations do not invalidate the matched eigenvalue comparisons or the feature-map freezing experiment, but weaken the broader interpretation of head specialization and fast weights relative to token-space memory. Repeated training seeds would also help establish the stability of the finer-grained ablation results, which currently use 64 evaluation episodes per length.

Questions for authors
  1. Clarification and potentially decisive control: Can you separate history loss from compositional failure in the causal Transformer evaluation—for example, by retaining the initial reveal and complete action history while controlling positional extrapolation? How much of the observed failure remains when task-relevant history is available?

  2. Mechanistic validation: Does the LaCT head-count trend persist at fixed backbone depth? Does normalizing or gating the scratchpad update prevent its reported norm explosion and improve extrapolation? These controls would materially affect the interpretation of which architectural properties are responsible.

  3. Scope of the claims: Can you distinguish claims established by prior asymptotic theory, hypotheses supported by the probes, and empirical findings specific to the tested training recipe? In particular, would you qualify “cannot carry” and “requires a revisable state” accordingly? Please also indicate whether the main ablation trends are stable across training seeds.

Reviewer_ay9JOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether video world models trained solely with video reconstruction supervision can track unobserved states, using an action-conditioned Shell Game that separately evaluates visual fidelity and hidden-state accuracy. Models trained on short swap sequences often fail to generalize to longer sequences, whereas linear attention with negative transition eigenvalues and nonlinear fast-weight updates substantially improve length extrapolation. Experiments on Memory Maze and Block World further explore observation-dependent state transitions and autonomous out-of-view dynamics.

Critical strengths
  1. A valuable controlled evaluation of hidden-state tracking. The visual Shell Game provides an interpretable setting in which plausible video generation can coexist with incorrect hidden-state predictions. Separately evaluating visual quality and final ball-position accuracy, together with length extrapolation, makes the distinction between these capabilities particularly clear.

  2. Informative architectural comparisons and ablations. The paper compares a broad range of backbones and analyzes attention-window size, denoising steps, carried-state readouts, and fast-weight update mechanisms. In particular, the comparison between updating the nonlinear feature map and updating only the readout matrix provides concrete evidence about which components support extrapolation in this setting.

Critical weaknesses
  1. The incremental novelty over closely related work is limited. Prior works, including StEvo-Bench, MemoBench, and LiveWorld, have investigated unobserved state evolution and the limitations of static visual memory. The visual Shell Game and its connection to state-tracking limitations also have a direct precedent in Liu and Lee (2026), although the present paper adapts the task from video understanding to video generation. Moreover, both successful mechanisms build on established results: Grazzi et al. analyze the role of negative eigenvalues in state tracking; LaCT already applies nonlinear fast weights to autoregressive video diffusion; and Liu et al. analyze history-dependent kernels induced by online feature-map updates. While the controlled video-generation experiments are useful, the paper should more clearly establish what new mechanistic understanding they provide beyond validating these existing ideas in a new setting.

  2. Several mechanistic interpretations are not fully isolated by the experiments. The LaCT head-count sweep also changes network depth: LaCT₁ uses eight layers, whereas LaCT₈ uses twelve layers (Table A2), making it difficult to attribute the improvement solely to head factorization. The authors report exploding memory norms in the scratchpad baseline (Section 5.3), but do not report ablations that explicitly normalize or gate the carried-state update. It therefore remains unclear whether its failure reflects the memory representation itself or the particular update implementation. Likewise, low linear-probe accuracy does not establish that hidden-state information is absent from the cache, particularly when the probes operate on a limited set of readouts rather than the full cache. Finally, the failure of GRU/LSTM variants to extrapolate in the video setting, despite their success on the symbolic task, shows that expressivity alone does not explain the results. Although Appendix G acknowledges this distinction, additional controlled experiments are needed to separate architectural limitations, optimization difficulties, and limitations of the training protocol.

Questions for authors

To what extent does the Transformer’s extrapolation failure arise from context truncation, as opposed to the proposed depth limitation? Footnote 4 states that the causal full-attention baseline is capped at the current chunk and seven previous chunks during longer rollouts, while unrestricted context introduces temporal RoPE offsets unseen during training. Figure A3 also acknowledges that truncation removes early chunks, including the initial reveal, creating a condition that SWA7 never encounters during training. Could the authors provide an additional experiment that retains the full relevant history while controlling positional-encoding extrapolation, to better isolate these factors?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.