- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper asks whether conventional world models, i.e., autoregressive and diffusion models built on transformers or convolutional networks, can learn the exact rules of a world from its observed history. The testbed is binary outer-totalistic cellular automata on an torus. A model observes 8 frames and must predict the next 8 exactly. Performance is measured by strict sequence accuracy: the fraction of rollouts in which all 512 predicted cells are correct. The authors attribute the failures of conventional models to three links. The first is spatial locality: identifying a cell's neighbourhood. The second is temporal locality: binding each observed situation to its outcome in the next frame. The third is temporal stability: settling each predicted frame before later frames are built on it. For each link, the authors propose a change that leaves the backbone untouched: two-dimensional rotary positions for spatial locality; a KV shift or a convolution over two adjacent frames for temporal locality; and causal freezing at sampling time for temporal stability. The paper also includes exploratory experiments on billiards and on several continuous dynamical systems.
Critical strengths
The authors use cellular automata as a testbed, a system whose rules are simple and fully known yet whose dynamics are rich. On this testbed they systematically test the mainstream world-model paradigms, namely autoregressive and diffusion models built on transformers or convolutional networks. Instead of relying on per-pixel accuracy, they require every cell of every predicted frame to be correct, which reveals where and how these models fail. The authors go beyond reporting failures and identify a common cause behind them: the models lack inductive biases matched to the task. They organize these inductive biases into three properties (spatial locality, temporal locality, and temporal stability) and give a corresponding fix for each. The controlled comparisons are also carefully designed; most of them change a single factor while holding everything else fixed.
Critical weaknesses
1. The writing and organization seriously hinder understanding. This is the main reason for my negative assessment.
(a) Most of the main text consists of self-contained paragraphs, each opening with a bold conclusion sentence, so the text effectively reads as a bulleted list. All four paragraphs of Section 3, most paragraphs of Sections 4.1–4.3, and all six paragraphs after the first in Section 5 open this way, and the introduction presents its contributions as three "Failure mode / Solution" boxes. This is more than a matter of style. Each paragraph states a conclusion and then its evidence, but the paper rarely says how one paragraph relates to the previous one: whether it adds to, qualifies, contradicts, or builds on it is left to the reader. As a result, the relations between the three properties, and between the experiments under each property, remain loose, and the paper reads as a list of findings rather than a coherent argument.
(b) Many concepts are used before they are defined, or are defined only in the appendix or in figure captions. For example, "situation" is already used in the introduction to state the temporal-locality failure but is only defined in Section 2. The levels L1–L4 appear in Figures 2 and 3 before they are defined in Section 2 and Table 1. The "structured diffusion model" is used to report results in Sections 4.2 and 4.3, but it is only defined at the end of Section 5. In addition, several terms are never explained in the main text, and the reader must look them up in the appendix or in figure captions: the "10% dose" in Section 4.3, the "canvas encoder" and "anchor frame" in Section 5, and width labels such as "d64" and "d128", which are used from Section 3 onward.
In my view, the paper needs substantial restructuring, not just local revision.
2. The paper adds limited new knowledge. The paper repeatedly emphasizes that adding the right inductive biases repairs the failures. However, the principle that architecture should align with the target computation is well known, and the authors themselves cite Battaglia et al. (2018) and Xu et al. (2020) on this point. The authors acknowledge that none of the devices they use (KV shift, causal freezing, 2D RoPE) is new. They position their contribution as providing a world in which the reason each device is needed can be stated in advance and tested. Yet the conclusions of these tests largely stay at the level of "whichever information pathway is missing, supply it through structure." After reading the paper, it is unclear to me what has been learned that was not known before and that transfers to other tasks.
The paper also does not place its experiments within the in-context learning framework. The contrast between L2 and L3 is essentially a distinction between in-weights and in-context learning. The task itself is associative recall, with the situation as the key and the next-frame state as the value. The authors cite in-context learning work such as Olsson et al. (2022) and Chan et al. (2022), but Chan et al. appears only in the L4 discussion of data dosage. Making this framing explicit would help readers see what the results add to existing understanding.
3. The paper offers no general methodology applicable to real-world world models. All three fixes rely on knowing the dependency structure of the rule in advance: the next state depends only on the 3×3 neighbourhood in the previous frame, and the rule is first-order in time. The checklist in Section 6 asks the reader to "write down which cells and frames must meet, and which parts of the future depend on which." For real-world world models, such as video generation or physical scene prediction, these dependencies are usually exactly what is unknown. Here the paper offers only a broad recommendation to attend to information and dependency structure. It does not give a concrete, actionable method for determining which information pathway needs to be supplied. The discussion of video models in Appendix H is likewise hypothetical. I understand that automatically discovering such pathways is beyond the scope of this paper. For exactly that reason, however, it is questionable how far its conclusions can guide the design of real-world world models.
Questions for authors
- The contrast between billiards and the Game of Life seems related to Wolfram's notion of computational irreducibility: the future positions of billiard balls have a closed-form solution, whereas the Game of Life must be evolved step by step. The authors give two explanations for this difference. First, the Game of Life cannot skip rule steps. Second, a later state can have several predecessors, so it cannot be used to verify an earlier guess. The first is close to irreducibility; the second is irreversibility. Could the authors compare joint sampling and causal freezing on rules with different degrees of reducibility and reversibility, to separate the contributions of these two factors?
- The held-out rules vary widely in difficulty. Under some rules, all cells quickly die or stay static; others are highly sensitive to perturbations. The paper reports only averages over all rules. Could the authors break down the results by rule type, for example by Wolfram class, Langton's λ parameter, or damage-spreading rate, to show whether the three properties play the same role across different kinds of rules?