- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Video world models generate fidelity scenes but may not maintain the hidden state of the world. This paper uses an action-conditioned video shell game as the testbed. Experiments show that standard backbones could render plausible videos but fail to generalize to longer swap chains. The authors point out that the pixel-based diffusion loss does not force the generated frames to hold the unseen ball position. The authors also note that in causal transformer, the implicit state is an append-only KV cache, for which the model must re-compute the swaps at every chunk. Then, the paper studies what enables learning the rule. The finding is that length generalization requires a revisable state carried across chunks and an update expressive enough to apply a swap.
Critical strengths
-
The paper studies a critial but often omitted problem: the state transition in a scene. It provides insights on building more stateful world models. Beyond controllability or fidelity, enabling state transition tracking is also important for world models.
-
The experiment uses a clear and meaningful diagnostic task, Shell Game, which separate video generation and state tracing, making it possible to study the hidden-states in different model types. The accompanying symbolic experiments help distinguish difficulties in composing state transitions from those introduced by visual learning. A shared video backbone and approximately matched parameter budgets further improve the interpretability of the architectural comparisons.
-
The evaluation spans Transformers, recurrent networks, state-space models, linear attention, and nonlinear fast weights. Beyond comparing overall performance, the paper examines sequence length, attention windows, denoising steps, transition eigenvalues, fast-weight heads, and which parameters receive online updates. These complementary experiments address several plausible explanations for success and failure.
Critical weaknesses
-
The Shell Game isolates action-driven permutation tracking effectively, but provides limited evidence about state estimation that requires integrating observations and correcting prior beliefs. Real world tasks are more complex.
-
The additional benchmarks provide limited direct validation of hidden-state tracking. Their evaluation primarily measures visual reconstruction quality, leaving it unclear whether improvements reflect more accurate state maintenance or better retrieval and rendering.
Questions for authors
None