- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3(+2)
Read review
Paper summary
The paper introduces a novel world-model architecture, GridWM, that handles long contexts and long rollouts in spatial environments under bounded memory budgets. Instead of stacking identical transformer layers, GridWM separates them into temporal layers followed by spatial layers, which positionally embed a sequence of historical observations using regular RoPE (i.e., "time RoPE") and grid-like path-integrated RoPE (i.e., "spatial RoPE"), respectively. The two layers also differ in memory organization: the temporal layer uses a sliding window over recent tokens, while the spatial layer attends to recent tokens + an external KV cache of earlier temporal-layer-embedded memories. A learned content-conditioned rotational offset is added to spatial RoPE to improve memory retrieval in the scenario where a single observation sees a combination of objects located at different locations. Experiments show that GridWM outperforms baselines under long rollouts and long contexts. Ablations validate that removing or changing their proposed architecture degrades performance. Broadly, this paper demonstrates how having segregated design across layers, when combined with inductive biases, can improve world models' generalization under bounded memories.
Critical strengths
The paper argues for stacking specialized layers rather than uniform layers, and this is supported by their ablations (Fig 3). This is insightful and may encourage research in world models with heterogeneous layers.
Critical weaknesses
Major weaknesses
1. Unclear methodology (Sec 2)
- I recommend swapping sec 2.2 and sec 2.1. Explaining technical details before establishing the overall objective, the input/output, and architecture is confusing.
- Sec 2.2 is unclear. What constitutes a token (L151)? What are “observation memories” (L152)? I recommend starting with notation definitions, then clearly defining the input and output of each component using them.
- The mathematical formulation of “temporal RoPE” is never formally defined. I assumed it was regular RoPE. I suggest clarifying this.
- L174-175 says the training is done on chunks of 16 frames, while L191 says that “All models are trained on continuous 320-frame trajectories.”, Which one is it?
- Sec 2.4, the definition of the loss should be in the main methods since they are crucial
- Fig. 1 is confusing. Shouldn’t the input action be (matching L142)? Shouldn’t the model also take in all further actions from to ? What is a coordinate transformer- another transformer model? is never formally defined. Is it the same or different from p? Temporal transformer and spatial transformer also differ in memory organization; I suggest also visualizing this. Even though the factorization design is properly highlighted, the unclear details make it difficult to understand the methods.
- The motion encoder, which is essential for path integration, should be in the main paper rather than the appendix.
2. The benefit of using 120 deg separated wave vector directions is unclear
- Fig 3E and the paragraph starting from L400. The description of “Cartesian-coordinate RoPE” and “single-period three-direction encoding” is missing. They’re important to judge and isolate the benefit of using grid-like RoPE. I assumed the latter is spatial-RoPE, except you would use a single across token dimensions.
- What’s the unique benefit of using 120 deg separated wave vector directions? Have the authors compared it with 3 vectors with random directions (rather than grid-cell-like hexagonal)? 4 directions? I think such a comparison is important because the paper title highlighted “grid-like spatial indexing”. Does a grid-cell-like hexagonal prior really matter?
3. Insufficient evidence supporting the benefit of content-conditioned spatial addressing.
I understand the motivation behind the design, but the evidence does not fully support it addresses the intended problem.
- Fig 4a), the comparison between GridWM and No Offset is insightful. However, do GridWM No Offset match in capacity (i.e number of parameters)? Otherwise, it's hard to tell whether the proposed design is merely adding capacity.
- The experimental description of Fig 4b) is unclear. What’s the exact procedure for measuring the distance? I assumed the authors are taking the sum of $\Delta{p_t}$ over $t$ as the path-integrated position. Isn’t this term dependent on a learned component (i.e $\hat{s_t}$ is predicted). If so, how do you ensure that $\Delta{p_t}$ is interpretable and corresponds to physical distance? How are the results aggregated over different heads?
- The experimental design of Fig 4b) is unjustified. Why measure the distance to the wall intersected by the current viewing direction and not all observed walls (e.g when multiple walls are observed, like in Fig. 4c)? Are the distance to the wall and the direction to the wall controlled?
- Fig 4b) reported that most offsets moved closer to the wall. What’s the distribution of the magnitude and direction of movement? Do they differ across heads? This is important to judge whether different heads learned distinct and meaningful spatial offsets.
- Fig 4b) lacks baselines. In particular, the current view is already facing the target wall, so a trivial move-forward offset can produce the same result. Baselines such as random offset and constant forward offset should be compared to establish that the learned offset actually “aligns with visible scene geometry”, rather than merely reflecting content-independent movement bias (e.g training trajectory primarily involves moving forward).
- Fig 4c) failed to demonstrate the proposed retrieval mechanism. First, L458 claims that the two memories receive high attention scores but fails to compare it with the attention scores assigned to other memories. Second, the number and the content of the remaining memories are not shown. Are they controlled? Third, a high attention score doesn’t mean the model is retrieving from this specific memory. Shuffling the retrieval results of H0 or H2 is too coarse to demonstrate the importance of the two memories shown in fig4c). The authors should show that selectively replacing the retrieved content of these two memories impairs the model’s prediction. Lastly and most importantly, the No Offset baseline from Fig 4a) is not compared here. Without this comparison, such observed retrieval behaviour cannot be attributed to their proposed content-conditioned offset.
Minor weaknesses
1. Limited methodological novelty
The grid-like RoPE highly resembles those of prior work [1,2]. The main contribution appears to be the combination of existing spatial positional encodings with a hierarchical temporal–spatial Transformer, bounded long-term KV memory, and content-conditioned offsets. Furthermore, the benefit of content-conditioned offsets is not convincing due to the aforementioned issues.
2. Pairwise Loss (sec 2.4) is unclear and not sufficiently justified
First, the and in Sec A.4 is unexplained. Their corresponding threshold values are also unjustified. Second, there’s not ablation of this loss term. How important is this loss term? Third, the heuristic that visual similarity + head direction similarity ~= positional similarity is questionable, which makes the loss term's ablation more important.
References:
[1] A grid cell-inspired unified position embedding for arbitrary-dimensional spaces. Li et al.
[2] nD-RoPE: A Generalized RoPE for n-Dimensional Position Embedding. Li et al.
Questions for authors
See the weakness section.
4 comments
Spatial Encoding, Methodological Novelty, and Pairwise Loss
W2. Clarifying the Role of 120° Plane Waves and the Novelty Boundary.
Appendix F.1 now defines three dimension-matched encodings—Grid-like, Cartesian, and Single-period—and details their periods, wave directions, and path-integration settings.
We agree that the current evidence does not establish a universal, independent performance advantage for hexagonal wave directions. We have therefore corrected the overly strong claims and positioned Grid-like encoding as a biologically and theoretically motivated default with robust empirical performance.
Additional analyses in Appendix F show that multi-periodic encoding achieves lower prediction errors than Single-period at greater distances from the last observation. Cross-encoding memory-attention comparisons also show that Grid-like assigns slightly more attention to memories with nearby positions and matching headings (6.81% vs. 6.51%/5.70%).
Our primary contributions are hierarchical spatiotemporal organization and content-conditioned spatial addressing, whose effectiveness does not depend on a unique advantage of hexagonal encoding.
Minor W1. Methodological Novelty.
We reorganized the Introduction to clarify that Grid-like RoPE serves as a default spatial encoding. Our core contributions are:
- Spatiotemporal organization: Systematic ablations demonstrate that temporal integration followed by spatial retrieval outperforms temporal-only and joint spatiotemporal encodings, with consistent gains as Transformer depth increases (Fig. 3).
- Content-adaptive addressing: A lightweight mechanism adapts spatial attention addresses to represented content, yielding prediction gains and interpretable geometric patterns (Fig. 4, Appendix G).
We appreciate your recognition of the value of heterogeneous layer organization. The revision further clarifies these two contributions and their supporting evidence.
Minor W2. Pairwise Spatial-Representation Loss.
We added explicit definitions of visual similarity, spatial similarity, and the loss function to the main text, together with its supervision objective. The loss applies only to observed trajectories and constructs weak spatial supervision from visual and heading similarities, without requiring ground-truth positions.
The thresholds are empirical criteria for selecting high-confidence positive and negative pairs rather than precise physical position labels. This loss supervises the visual-action motion encoder, whose motion information is subsequently transferred to the action-only branch through the teacher–student loss.
The new Appendix Fig. 11 shows an approximately linear relationship between the action-only motion encoder's predicted displacements and ground-truth motion, providing further evidence that the training scheme learns meaningful motion representations.
We thank you again for your detailed feedback. We hope the revised methodology and additional mechanistic analyses more clearly establish GridWM's implementation, core contributions, and empirical support.
reply to author's rebuttal
I appreciate the author's detailed response, additional experiments, and substantial revision to the manuscript. All my weaknesses have been resolved, and I'm therefore happy to increase my rating to 3. However, I still have a few more clarification questions and suggestions.
Clarification questions:
- What's the action space? (i.e what constitutes ? Is it a vector displacement?) Also, doesn't action already entail the displacement ? Why do you need a learnable convolution layer to predict from ?
Suggestions:
-
As pointed out by me and other reviewers, the original manuscript emphasizes too much on grid-like RoPE without it being the core innovation nor properly investigated. The revised manuscript addressed this to some extent. However, I would still suggest that the authors consider avoiding "grid" in the model name and in the title, so as not to leave a wrong expectation for future readers.
-
I think Fig.12 in the appendix demonstrates the benefit of content-adaptive addressing substantially better than Fig4b), c). I suggest moving some results from Fig.12 to the main paper.
-
Since Content-adaptive addressing is a core innovation, I suggest demonstrating the problem it tries to solve either with a schematic (e.g add to Fig1) or a concrete example (e.g add to Sec 2.3). e.g., for a schematic, a top-down view showing that the views that need to be predicted are observed across separate memories will illustrate the problem much more clearly. It wasn't clear to me what problem it tries to solve until sec 4.3
We sincerely thank you for your encouraging feedback and for reconsidering your assessment. We are particularly pleased that our revisions and additional experiments have resolved your previous concerns. We also appreciate your further constructive suggestions.
Action space and motion prediction. The action input represents an environment control command rather than a measured displacement. In MemoryMaze, actions are discrete navigation commands, while HM3D uses forward movements and turns. Importantly, commanded actions do not always produce the intended displacement due to collisions (4.4% of actions in our MemoryMaze dataset). The motion encoder therefore estimates displacement from action history, with visual observations providing additional supervision during training. The action-only branch enables path integration during open-loop prediction, although localization drift can still accumulate, as analyzed in Appendix B.1.
Presentation improvements. We agree that the title and model name should better reflect the central contributions of hierarchical spatiotemporal organization and content-adaptive addressing, and will carefully reconsider the naming in the final manuscript. We also appreciate your suggestions to bring the key findings from Appendix Fig. 12 into the main text and add a motivating illustration or example to Section 2.3. We plan to incorporate these improvements in the final version.
Thank you again for your thoughtful and constructive engagement. Your suggestions have been valuable in helping us communicate the contributions of this work more clearly.
Content-Conditioned Addressing and Methodological Clarity
We thank you for recognizing GridWM's hierarchical architectural design and supporting ablations. Your detailed suggestions on methodological clarity and content-conditioned addressing have helped us clarify the implementation and strengthen the empirical evidence for our proposed mechanism.
W3. Strengthening the Mechanistic Evidence for Content-Conditioned Spatial Addressing.
We appreciate your specific suggestions. Building on the original Fig. 4 ablations, we added Appendix G to further investigate learned offsets through parameter efficiency, spatial geometry, and targeted memory interventions.
(1) Performance gains reflect meaningful address adaptation (W3.1, 5). The offset module adds only 0.051M parameters (0.67%), yet reduces LPIPS by 9.00% and 5.56% under short- and long-horizon prediction, respectively. Moreover, the original Fig. 4A already includes test-time interventions using mean, shuffled, and zero offsets, all of which perform worse than the full model. These controls indicate that the gains cannot be explained solely by additional capacity or a fixed forward bias.
(2) Learned offsets exhibit interpretable spatial structure (W3.2–4). Spatial phases are obtained through path integration of predicted displacements, with no learnable parameters in the phase-update operation. The new Appendix G.1 shows an approximately linear relationship between predicted and ground-truth displacements.
In the original Fig. 4B, we use environment geometry to determine the wall intersection along the current viewing direction and find that 92.4% of adjusted addresses move closer to the corresponding wall. Addressing your questions about directional preferences and head specialization, Appendix G.2 further analyzes all attention heads. Most heads exhibit relatively stable, distributed directional preferences across the approximately 80° field of view. Offset magnitudes correlate positively with nearest-wall distances along their preferred directions, particularly within three environment cells, before gradually saturating. These findings further support the geometric adaptivity of learned offsets.
(3) Targeted memory interventions validate retrieval effects (W3.6). The new Appendix Fig. 12C–E examines the case in Fig. 4C in greater detail, showing all historical frames receiving over 10% attention from H0 or H2 and their corresponding spatial addresses. H0 primarily attends to the green wall, while H2 focuses on the red wall. The full model substantially outperforms No Offset (LPIPS: 0.114 vs. 0.251). Replacing the high-attention slots of H0 or H2 primarily affects predictions of the green or red wall, respectively, while intervening on both causes larger errors.
Together, these results show that content-conditioned offsets learn geometrically meaningful, head-specific addressing patterns with minimal parameter overhead, improving visual prediction through selective use of historical experiences.
W1. Clarifying Methodology and Implementation Details.
We carefully revised Section 2 to address all seven concerns:
We also standardized the notation and data flow. The complete implementation and configurations are available in our anonymous repository.