acceptodds
Under review as a conference paper at ICLR 2027

GridWM: Grid-Like Spatial Indexing for Long-Horizon World Models

Abstract

Long-horizon world modeling in partially observable environments requires compact memories that preserve spatial structure despite accumulated localization drift. Inspired by the multi-periodic coding of entorhinal grid cells, we propose GridWM, which adopts a layer-wise organization of local temporal RoPE and multi-periodic spatial RoPE, together with content-conditioned address offsets, to capture recent temporal context and long-range spatial structure. A fixed-capacity memory keeps KV storage and per-step attention cost bounded as the sequence grows. On MemoryMaze, GridWM outperforms strong world-model baselines within the training sequence length and generalizes substantially better to prediction horizons far beyond those seen during training, while using a compact model. The temporal-spatial encoding remains effective across deeper Transformer stacks, with consistent gains as depth increases. These results show that combining local temporal context with structured spatial indexing provides an effective inductive bias for efficient long-horizon world modeling.

Primary area:
applications to neuroscience & cognitive science
Keywords:
world models, spatial memory, grid cells, long-horizon prediction, rotary positional encoding

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_RFANOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+2)
Read review
Paper summary

​​The paper introduces a novel world-model architecture, GridWM, that handles long contexts and long rollouts in spatial environments under bounded memory budgets. Instead of stacking identical transformer layers, GridWM separates them into temporal layers followed by spatial layers, which positionally embed a sequence of historical observations using regular RoPE (i.e., "time RoPE") and grid-like path-integrated RoPE (i.e., "spatial RoPE"), respectively. The two layers also differ in memory organization: the temporal layer uses a sliding window over recent tokens, while the spatial layer attends to recent tokens + an external KV cache of earlier temporal-layer-embedded memories. A learned content-conditioned rotational offset is added to spatial RoPE to improve memory retrieval in the scenario where a single observation sees a combination of objects located at different locations. Experiments show that GridWM outperforms baselines under long rollouts and long contexts. Ablations validate that removing or changing their proposed architecture degrades performance. Broadly, this paper demonstrates how having segregated design across layers, when combined with inductive biases, can improve world models' generalization under bounded memories.

Critical strengths

The paper argues for stacking specialized layers rather than uniform layers, and this is supported by their ablations (Fig 3). This is insightful and may encourage research in world models with heterogeneous layers.

Critical weaknesses

Major weaknesses

1. Unclear methodology (Sec 2)

  1. I recommend swapping sec 2.2 and sec 2.1. Explaining technical details before establishing the overall objective, the input/output, and architecture is confusing.
  2. Sec 2.2 is unclear. What constitutes a token (L151)? What are “observation memories” (L152)? I recommend starting with notation definitions, then clearly defining the input and output of each component using them.
  3. The mathematical formulation of “temporal RoPE” is never formally defined. I assumed it was regular RoPE. I suggest clarifying this.
  4. L174-175 says the training is done on chunks of 16 frames, while L191 says that “All models are trained on continuous 320-frame trajectories.”, Which one is it?
  5. Sec 2.4, the definition of the loss should be in the main methods since they are crucial
  6. Fig. 1 is confusing. Shouldn’t the input action be at+1a_{t+1} (matching L142)? Shouldn’t the model also take in all further actions from at+1a_{t+1} to at+Ha_{t+H}? What is a coordinate transformer- another transformer model? GtG_t is never formally defined. Is it the same or different from Δ\Deltap? Temporal transformer and spatial transformer also differ in memory organization; I suggest also visualizing this. Even though the factorization design is properly highlighted, the unclear details make it difficult to understand the methods.
  7. The motion encoder, which is essential for path integration, should be in the main paper rather than the appendix.

2. The benefit of using 120 deg separated wave vector directions is unclear

  1. Fig 3E and the paragraph starting from L400. The description of “Cartesian-coordinate RoPE” and “single-period three-direction encoding” is missing. They’re important to judge and isolate the benefit of using grid-like RoPE. I assumed the latter is spatial-RoPE, except you would use a single λg\lambda_g across token dimensions.
  2. What’s the unique benefit of using 120 deg separated wave vector directions? Have the authors compared it with 3 vectors with random directions (rather than grid-cell-like hexagonal)? 4 directions? I think such a comparison is important because the paper title highlighted “grid-like spatial indexing”. Does a grid-cell-like hexagonal prior really matter?

3. Insufficient evidence supporting the benefit of content-conditioned spatial addressing.

I understand the motivation behind the design, but the evidence does not fully support it addresses the intended problem.

  1. Fig 4a), the comparison between GridWM and No Offset is insightful. However, do GridWM No Offset match in capacity (i.e number of parameters)? Otherwise, it's hard to tell whether the proposed design is merely adding capacity.
  2. The experimental description of Fig 4b) is unclear. What’s the exact procedure for measuring the distance? I assumed the authors are taking the sum of $\Delta{p_t}$ over $t$ as the path-integrated position. Isn’t this term dependent on a learned component (i.e $\hat{s_t}$ is predicted). If so, how do you ensure that $\Delta{p_t}$ is interpretable and corresponds to physical distance? How are the results aggregated over different heads?
  3. The experimental design of Fig 4b) is unjustified. Why measure the distance to the wall intersected by the current viewing direction and not all observed walls (e.g when multiple walls are observed, like in Fig. 4c)? Are the distance to the wall and the direction to the wall controlled?
  4. Fig 4b) reported that most offsets moved closer to the wall. What’s the distribution of the magnitude and direction of movement? Do they differ across heads? This is important to judge whether different heads learned distinct and meaningful spatial offsets.
  5. Fig 4b) lacks baselines. In particular, the current view is already facing the target wall, so a trivial move-forward offset can produce the same result. Baselines such as random offset and constant forward offset should be compared to establish that the learned offset actually “aligns with visible scene geometry”, rather than merely reflecting content-independent movement bias (e.g training trajectory primarily involves moving forward).
  6. Fig 4c) failed to demonstrate the proposed retrieval mechanism. First, L458 claims that the two memories receive high attention scores but fails to compare it with the attention scores assigned to other memories. Second, the number and the content of the remaining memories are not shown. Are they controlled? Third, a high attention score doesn’t mean the model is retrieving from this specific memory. Shuffling the retrieval results of H0 or H2 is too coarse to demonstrate the importance of the two memories shown in fig4c). The authors should show that selectively replacing the retrieved content of these two memories impairs the model’s prediction. Lastly and most importantly, the No Offset baseline from Fig 4a) is not compared here. Without this comparison, such observed retrieval behaviour cannot be attributed to their proposed content-conditioned offset.

Minor weaknesses

1. Limited methodological novelty

The grid-like RoPE highly resembles those of prior work [1,2]. The main contribution appears to be the combination of existing spatial positional encodings with a hierarchical temporal–spatial Transformer, bounded long-term KV memory, and content-conditioned offsets. Furthermore, the benefit of content-conditioned offsets is not convincing due to the aforementioned issues.

2. Pairwise Loss (sec 2.4) is unclear and not sufficiently justified

First, the VijV_{ij} and SijS_{ij} in Sec A.4 is unexplained. Their corresponding threshold values are also unjustified. Second, there’s not ablation of this loss term. How important is this loss term? Third, the heuristic that visual similarity + head direction similarity ~= positional similarity is questionable, which makes the loss term's ablation more important.

References:

[1] A grid cell-inspired unified position embedding for arbitrary-dimensional spaces. Li et al.

[2] nD-RoPE: A Generalized RoPE for n-Dimensional Position Embedding. Li et al.

Questions for authors

See the weakness section.

4 comments
AuthorsOpenReview ↗

Content-Conditioned Addressing and Methodological Clarity

We thank you for recognizing GridWM's hierarchical architectural design and supporting ablations. Your detailed suggestions on methodological clarity and content-conditioned addressing have helped us clarify the implementation and strengthen the empirical evidence for our proposed mechanism.

W3. Strengthening the Mechanistic Evidence for Content-Conditioned Spatial Addressing.

We appreciate your specific suggestions. Building on the original Fig. 4 ablations, we added Appendix G to further investigate learned offsets through parameter efficiency, spatial geometry, and targeted memory interventions.

(1) Performance gains reflect meaningful address adaptation (W3.1, 5). The offset module adds only 0.051M parameters (0.67%), yet reduces LPIPS by 9.00% and 5.56% under short- and long-horizon prediction, respectively. Moreover, the original Fig. 4A already includes test-time interventions using mean, shuffled, and zero offsets, all of which perform worse than the full model. These controls indicate that the gains cannot be explained solely by additional capacity or a fixed forward bias.

(2) Learned offsets exhibit interpretable spatial structure (W3.2–4). Spatial phases are obtained through path integration of predicted displacements, with no learnable parameters in the phase-update operation. The new Appendix G.1 shows an approximately linear relationship between predicted and ground-truth displacements.

In the original Fig. 4B, we use environment geometry to determine the wall intersection along the current viewing direction and find that 92.4% of adjusted addresses move closer to the corresponding wall. Addressing your questions about directional preferences and head specialization, Appendix G.2 further analyzes all attention heads. Most heads exhibit relatively stable, distributed directional preferences across the approximately 80° field of view. Offset magnitudes correlate positively with nearest-wall distances along their preferred directions, particularly within three environment cells, before gradually saturating. These findings further support the geometric adaptivity of learned offsets.

(3) Targeted memory interventions validate retrieval effects (W3.6). The new Appendix Fig. 12C–E examines the case in Fig. 4C in greater detail, showing all historical frames receiving over 10% attention from H0 or H2 and their corresponding spatial addresses. H0 primarily attends to the green wall, while H2 focuses on the red wall. The full model substantially outperforms No Offset (LPIPS: 0.114 vs. 0.251). Replacing the high-attention slots of H0 or H2 primarily affects predictions of the green or red wall, respectively, while intervening on both causes larger errors.

Together, these results show that content-conditioned offsets learn geometrically meaningful, head-specific addressing patterns with minimal parameter overhead, improving visual prediction through selective use of historical experiences.

W1. Clarifying Methodology and Implementation Details.

We carefully revised Section 2 to address all seven concerns:

  • Architecture and notation (W1.1–2): We reorganized the section to introduce the model's inputs, outputs, and overall spatiotemporal pipeline before presenting spatial encoding details. We simplified the descriptions of tokens and observation memories, clarified the per-step visual-action inputs and the roles of temporal and spatial layers, and specified that spatial layers access KV caches from the 32 most recent steps and up to 96 earlier steps.
  • Positional encoding and coordinate transformation (W1.3, 6–7): We clarified that temporal RoPE is standard RoPE and added its original citation. Figure 1 now consistently uses the next action at+1a_{t+1}, with the spatial state denoted by ctc_t. The Coordinate Transformer performs a deterministic egocentric-to-world transformation using ground-truth heading, without learnable parameters. We also moved the essential motion encoder description into the main text.
  • Training and prediction (W1.4–5, 6): We clarified that training uses 320-frame sequences with alternating 16-frame observation and open-loop prediction chunks. During inference, future actions are provided sequentially, and predicted representations are reused for autoregressive rollout. Definitions of all four training losses are now included in the main text.

We also standardized the notation and data flow. The complete implementation and configurations are available in our anonymous repository.

AuthorsOpenReview ↗

Spatial Encoding, Methodological Novelty, and Pairwise Loss

W2. Clarifying the Role of 120° Plane Waves and the Novelty Boundary.

Appendix F.1 now defines three dimension-matched encodings—Grid-like, Cartesian, and Single-period—and details their periods, wave directions, and path-integration settings.

We agree that the current evidence does not establish a universal, independent performance advantage for hexagonal wave directions. We have therefore corrected the overly strong claims and positioned Grid-like encoding as a biologically and theoretically motivated default with robust empirical performance.

Additional analyses in Appendix F show that multi-periodic encoding achieves lower prediction errors than Single-period at greater distances from the last observation. Cross-encoding memory-attention comparisons also show that Grid-like assigns slightly more attention to memories with nearby positions and matching headings (6.81% vs. 6.51%/5.70%).

Our primary contributions are hierarchical spatiotemporal organization and content-conditioned spatial addressing, whose effectiveness does not depend on a unique advantage of hexagonal encoding.

Minor W1. Methodological Novelty.

We reorganized the Introduction to clarify that Grid-like RoPE serves as a default spatial encoding. Our core contributions are:

  1. Spatiotemporal organization: Systematic ablations demonstrate that temporal integration followed by spatial retrieval outperforms temporal-only and joint spatiotemporal encodings, with consistent gains as Transformer depth increases (Fig. 3).
  2. Content-adaptive addressing: A lightweight mechanism adapts spatial attention addresses to represented content, yielding prediction gains and interpretable geometric patterns (Fig. 4, Appendix G).

We appreciate your recognition of the value of heterogeneous layer organization. The revision further clarifies these two contributions and their supporting evidence.

Minor W2. Pairwise Spatial-Representation Loss.

We added explicit definitions of visual similarity, spatial similarity, and the loss function to the main text, together with its supervision objective. The loss applies only to observed trajectories and constructs weak spatial supervision from visual and heading similarities, without requiring ground-truth positions.

The thresholds are empirical criteria for selecting high-confidence positive and negative pairs rather than precise physical position labels. This loss supervises the visual-action motion encoder, whose motion information is subsequently transferred to the action-only branch through the teacher–student loss.

The new Appendix Fig. 11 shows an approximately linear relationship between the action-only motion encoder's predicted displacements and ground-truth motion, providing further evidence that the training scheme learns meaningful motion representations.

We thank you again for your detailed feedback. We hope the revised methodology and additional mechanistic analyses more clearly establish GridWM's implementation, core contributions, and empirical support.

Reviewer_RFANOpenReview ↗

reply to author's rebuttal

I appreciate the author's detailed response, additional experiments, and substantial revision to the manuscript. All my weaknesses have been resolved, and I'm therefore happy to increase my rating to 3. However, I still have a few more clarification questions and suggestions.

Clarification questions:

  1. What's the action space? (i.e what constitutes ata_t? Is it a vector displacement?) Also, doesn't action ata_t already entail the displacement st^\hat{s_t}? Why do you need a learnable convolution layer to predict st^\hat{s_t} from ata_t?

Suggestions:

  1. As pointed out by me and other reviewers, the original manuscript emphasizes too much on grid-like RoPE without it being the core innovation nor properly investigated. The revised manuscript addressed this to some extent. However, I would still suggest that the authors consider avoiding "grid" in the model name and in the title, so as not to leave a wrong expectation for future readers.

  2. I think Fig.12 in the appendix demonstrates the benefit of content-adaptive addressing substantially better than Fig4b), c). I suggest moving some results from Fig.12 to the main paper.

  3. Since Content-adaptive addressing is a core innovation, I suggest demonstrating the problem it tries to solve either with a schematic (e.g add to Fig1) or a concrete example (e.g add to Sec 2.3). e.g., for a schematic, a top-down view showing that the views that need to be predicted are observed across separate memories will illustrate the problem much more clearly. It wasn't clear to me what problem it tries to solve until sec 4.3

We sincerely thank you for your encouraging feedback and for reconsidering your assessment. We are particularly pleased that our revisions and additional experiments have resolved your previous concerns. We also appreciate your further constructive suggestions.

Action space and motion prediction. The action input represents an environment control command rather than a measured displacement. In MemoryMaze, actions are discrete navigation commands, while HM3D uses forward movements and turns. Importantly, commanded actions do not always produce the intended displacement due to collisions (4.4% of actions in our MemoryMaze dataset). The motion encoder therefore estimates displacement from action history, with visual observations providing additional supervision during training. The action-only branch enables path integration during open-loop prediction, although localization drift can still accumulate, as analyzed in Appendix B.1.

Presentation improvements. We agree that the title and model name should better reflect the central contributions of hierarchical spatiotemporal organization and content-adaptive addressing, and will carefully reconsider the naming in the final manuscript. We also appreciate your suggestions to bring the key findings from Appendix Fig. 12 into the main text and add a motivating illustration or example to Section 2.3. We plan to incorporate these improvements in the final version.

Thank you again for your thoughtful and constructive engagement. Your suggestions have been valuable in helping us communicate the contributions of this work more clearly.

Reviewer_9P4yOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes GridWM, a hierarchical Transformer-based world model for long-horizon visual prediction. It combines local temporal RoPE, multi-periodic grid-like spatial RoPE, content-conditioned address offsets, and fixed-capacity KV memory to organize temporal context and long-range spatial information. Experiments on MemoryMaze demonstrate improvements over several baselines, especially beyond the training horizon. The paper also provides ablation studies and preliminary results on HM3D.

Critical strengths
  1. Strong performance on long-horizon visual prediction.

GridWM consistently outperforms several competitive baselines on MemoryMaze, particularly under the challenging 500 →\rightarrow 500 prediction protocol. The improvements across multiple evaluation metrics demonstrate the effectiveness of the proposed architecture.

  1. Interesting architectural design with comprehensive ablations.

The separation of local temporal processing and long-range spatial retrieval is intuitive and well motivated. The ablations on positional encodings, layer arrangements, memory management, and address offsets provide useful insights into designing memory-efficient world models.

Critical weaknesses
  1. The central motivation and technical contributions are not sufficiently focused.

The paper introduces several ideas, including grid-like spatial encoding, hierarchical temporal-spatial attention, and content-conditioned address offsets. While each component is individually reasonable, their relationships to a unified research question are not clearly established. In particular, the necessity of grid-like encoding remains unclear, as simpler spatial representations achieve relatively comparable performance under the longest prediction horizon. The main contribution appears to be the architectural organization rather than the grid-like representation itself.

  1. Insufficient analysis of the grid-like spatial representation.

Although grid-like spatial indexing is a central component of the proposed method, the paper provides limited analysis of the representation itself. The comparison with Cartesian and single-period encodings in Figure 3E relies entirely on downstream image-prediction metrics, without examining spatial discriminability, periodicity, or memory retrieval accuracy. Consequently, it remains unclear what specific advantages the multi-periodic grid-like representation provides beyond improving prediction performance.

Questions for authors
  1. What is the primary contribution of GridWM? Is it the grid-like spatial encoding, the hierarchical attention architecture, or the content-conditioned addressing mechanism? Could the authors clarify how these components contribute to a unified research objective?

  2. What specific properties of the multi-periodic grid-like representation contribute to improved prediction? Could the authors provide more direct analyses of spatial discriminability, periodicity, or robustness to spatial ambiguity, rather than relying primarily on image-prediction metrics?

  3. Does grid-like spatial encoding actually improve long-term memory retrieval? Could the authors quantitatively evaluate retrieval accuracy or the relationship between spatial proximity and attention weights, compared with alternative spatial encodings?

4 comments

We thank you for recognizing GridWM's strong long-horizon prediction performance, well-motivated hierarchical design, and comprehensive ablations. Your feedback has helped us sharpen our core contributions and strengthen the analyses of spatial representations and memory retrieval.

W1 & Q1. Clarifying Core Contributions and Research Focus.

We agree with your assessment that architectural organization is the primary contribution. The original manuscript overemphasized grid-like encoding, obscuring our main research focus. We have rewritten the Introduction to clarify that our central question is how to effectively organize and utilize spatiotemporal information for long-horizon visual prediction. Our key contributions are:

  • Hierarchical spatiotemporal organization: Temporal-to-spatial processing outperforms temporal-only, spatial-only, and joint encodings, with consistent gains as Transformer depth increases (Fig. 3).
  • Content-conditioned spatial addressing: With only 0.051M additional parameters (0.67%), learned offsets reduce LPIPS by 9.00% and 5.56% under short- and long-horizon prediction, respectively, while exhibiting geometry-aligned, head-specific spatial patterns (Fig. 4, Appendix G).

Grid-like encoding serves as a biologically and theoretically motivated default, while the overall architecture remains robust across spatial encodings. We have corrected the overly strong claims about its independent advantages, without affecting our two core contributions or their experimental evidence.

W2 & Q2. Additional Theoretical and Empirical Analyses of Spatial Encoding.

Appendix F details the construction and controlled comparison of three dimension-matched spatial encodings, with further analyses of their properties.

  • Multi-periodic encoding: Previous theory supports multi-scale encoding for reducing long-range positional ambiguity. Our new Fig. 10 shows that the MSE and LPIPS gaps between Single-period and Grid-like generally increase with distance from the last observed position, consistent with this theoretical expectation.
  • Hexagonal geometry: Biological and theoretical studies motivate hexagonal encoding, but its advantages over orthogonal wave directions depend on specific conditions. Grid-like and Cartesian also achieve comparable performance in our experiments. We have therefore moderated our claims accordingly.
  • Spatial memory retrieval: Following your suggestion, we added a quantitative comparison of historical memory attention across the three encodings (Appendix F.2 Table9). On MemoryMaze under 256→64, Grid-like assigns 6.81% of observed-memory attention to nearby locations with matching headings, compared with 6.51% for Cartesian and 5.70% for Single-period. It also achieves a shorter attention-weighted spatial distance (2.93 vs. 3.00/3.12). These results indicate modest numerical advantages in spatially relevant retrieval, consistent with downstream prediction performance.

Together, these analyses provide theoretical and empirical support for multi-periodic grid-like encoding as a reliable default, while clarifying the limits of its advantages.

Q3. Long-Term Memory Retrieval.

We again apologize for overemphasizing encoding geometry without sufficiently highlighting how spatial information is utilized. GridWM's key innovation is to make spatial representations effective for long-term memory retrieval through hierarchical spatiotemporal processing and content-conditioned address offsets.

The original Fig. 4 already demonstrates the benefits of adaptive addressing through ablations and test-time interventions: removing or perturbing learned offsets degrades prediction, while 92.4% of adjusted addresses move closer to the corresponding walls.

The new Appendix G further reveals the underlying mechanisms. Different attention heads develop stable directional preferences, while offset magnitudes adapt to wall distances. Historical attention analysis and targeted memory interventions also show that complementary spatial cues retrieved by different heads contribute to the prediction of corresponding scene content.

Thus, cross-encoding comparisons reveal modest retrieval advantages for Grid-like encoding, while stronger ablation and mechanistic evidence supports the importance of content-conditioned addressing for long-term memory utilization. Our core contributions remain hierarchical spatiotemporal organization and adaptive spatial addressing.

Thank you again for your valuable suggestions. The revised research focus and additional analyses further clarify GridWM's technical contributions and empirical value.

Dear Reviewer,

Thank you for your time and thoughtful review. As the discussion period is approaching its end, we would greatly appreciate it if you could take a moment to review our rebuttal and additional experiments.

We would be happy to clarify any remaining concerns and look forward to your feedback.

Best regards, The Authors

Thank you for the detailed response. The revised positioning of the core contributions is much clearer, but I am still not fully convinced that the additional analyses directly demonstrate effective long-term spatial memory retrieval, rather than improvements in downstream prediction or attention distribution. Could the authors further clarify how the existing results specifically support the roles of hierarchical organization and content-conditioned addressing in memory retrieval? A clearer explanation would help me determine whether the remaining concerns are sufficiently addressed to reconsider my rating.

Thank you for acknowledging the clearer positioning of our core contributions. We understand your concern that prediction gains and attention distributions alone cannot directly establish effective long-term memory retrieval. We therefore clarify the mechanism and evidence through controlled ablations and targeted memory interventions.

1. How does hierarchical spatiotemporal organization support long-term retrieval?

GridWM assigns local experience integration and long-term spatial retrieval to separate Transformer layers. Temporal layers integrate visual and action information over a contiguous 32-step window into context-aware experience representations. Spatial layers use path-integrated spatial RoPE to incorporate relative spatial relationships into query–key attention, accessing a KV cache of the 32 most recent steps and up to 96 earlier entries. Attention can thus jointly exploit visual content and spatial relationships.

Fig. 3A–C isolates the architectural contribution with content-conditioned offsets disabled. Fig. 3A–B shows that temporal-then-spatial processing outperforms temporal-only, spatial-only, and within-layer joint encoding, with further gains as depth increases. Crucially, Fig. 3C shows that replacing the spatial layers’ long-term coverage cache with FIFO degrades performance; replacing the temporal layers’ contiguous cache with coverage sampling also hurts performance. These results support distinct memory-access requirements for the two stages under a fixed budget. Furthermore, Fig. 2C shows that longer observation histories improve the next-64-frame predictions even with a fixed 128-entry memory budget, supporting the model’s ability to use long-term context.

2. How do content-conditioned offsets improve spatial retrieval?

A single viewpoint-centered address cannot fully describe the spatial associations of visual features integrated across frames and scene regions. A lightweight MLP (0.051M parameters) therefore predicts a content-conditioned two-dimensional offset for each spatial attention head, allowing heads to adapt their retrieval addresses to different components of the represented content.

Appendix Fig. 12A–B shows relatively stable, distributed head-wise directional preferences across the approximately 80° field of view. Offset magnitude increases with the nearest-wall distance along each head’s preferred direction, most clearly within roughly three environment cells, before gradually saturating. These results support geometry-adaptive, head-specific addressing. Fig. 4A further confirms its contribution through No Offset, Mean Offset, and Shuffled Offset controls.

3. Targeted memory interventions directly test the use of long-term information.

Appendix Fig. 12C–E provides our most direct evidence. To limit direct reuse of previously seen views, we select a target whose position is over 4 m from every observed context position. In Fig. 4C and Appendix Fig. 12C–D, H0 and H2 attend strongly to historical frames #132 and #138, located 5.85 m and 4.40 m from the target, respectively, and both well beyond the temporal layer’s 32-step window. H0 attends mainly to observations of the distant green wall, while H2 attends to those of the left red wall. Neither historical view contains the complete target image, but they provide complementary cues.

Crucially, Appendix Fig. 12E tests whether these memories affect the prediction. The full model achieves 0.114 LPIPS. Replacing H0’s high-attention memory slots primarily disrupts the green-wall region (0.132 LPIPS); replacing H2’s primarily disrupts the red-wall region (0.148). Replacing both raises LPIPS to 0.175. The No Offset model exhibits visible errors in both walls (0.251).

These interventions directly connect head-specific spatial addressing, selected historical content, and changes in corresponding predicted regions. The model can combine complementary cues from spatially distinct historical observations even when the target view has never been fully observed.

In summary, Fig. 3A–C independently supports hierarchical processing and differentiated memory access; Fig. 4A and Appendix Fig. 12A–B support adaptive spatial addressing; and Appendix Fig. 12C–E provides intervention-based evidence beyond prediction scores and attention correlations that the full model uses relevant long-term experiences.

Although this case study does not replace a dataset-wide retrieval-accuracy benchmark, it supports our conclusion that hierarchical organization and content-conditioned addressing work together to enable effective long-term spatial memory utilization.

Thank you again for your careful engagement. We hope this clarifies both the retrieval mechanism and the evidence supporting our contributions.

Reviewer_B3oHOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
1
Soundness
3
Read review
Paper summary

Paper proposes GridWM, a hierarchical Transformer that combines temporal contextualization, multi-periodic grid-like spatial indexing, fixed-capacity memory, and content-conditioned address offsets. Inspired by the multi-periodic coding of entorhinal grid cells, the central claim is that it provides efficient access to long range history without unbounded storage and improves prediction within and beyond the training sequence length.

The training objective consists of an image-reconstruction loss, an open-loop prediction loss, a pairwise representation loss,and a teacher–student motion loss.

GridWM was evaluated on the open-loop long-horizon visual prediction task with the MemoryMaze benchmark under the 256→64 and 500→500 protocols. SOTA methods include reimplemented HPC–MEC, CoME trained with MemoryMaze, and constructed latent-space world models based on Transformer, RSSM, and Mamba. In both protocols, GridWM outperforms the compared methods significantly.

Ablation study comparing the temporal RoPE(T), spatial RoPE(S), within layer joint encoding(J), no positional encoding (N),and different layer-wise temporal–spatial arrangements shows that temporal and spatial positional structures provide complementary information. The results also show that spatial indexing allows the model to exploit information spanning long temporal ranges with fixed-capacity memory.

Critical strengths

Paper proposed a neural-inspired design for the fixed-capacity long-horizon memory retrieval problem with a hierarchical Transformer.

  1. The design appears to be novel, however, I am not up-to-date on this area of work.
  2. The experimental results support the hypothesis that the design allows for fixed-capacity memory with improved performance for the pair-wise perception similarity (LPIPS, SSIM, PSNR, MSE)
  3. Various ablation studies were done to analyse the contributions of the temporal, spatial arrangements and representation in the proposed design. The studies also supports the claim on the key contribution.
Critical weaknesses
  1. Experiment was only done on MemoryMaze benchmark which focus on next-frame prediction reconstruction.This limitation was identified and acknowledged in the paper.
  2. Novelty is limited as the temporal RoPE and spatial RoPE were commonly used in prior work. The spatial memory addressing methods were also proposed in similar prior works: Episodic Spatial World Model (ESWM, 2026); MemoryVLN (2026)
  3. (minor) The paper introduces several terms without proper references, e.g. RoPE

He, L., Daigle, K., & others. (2026). Building spatial world models from sparse transitional episodic memories. In International Conference on Learning Representations (ICLR 2026)

Hu, L., Zeng, H., Ma, X., Xue, M., Ying, Z., Kang, Z., Guo, Q., Zhang, J., & Liu, Y. (2026). MemoryVLN: Memory-augmented vision-language navigation. Transactions on Machine Learning Research.

Questions for authors

Please compare the ESWM and MemoryVLN memory design with your proposed work and highlight the novelty and advantages.

4 comments
AuthorsOpenReview ↗

Novelty, HM3D Evaluation, and Related Work

Thank you for recognizing GridWM's long-horizon prediction capability, fixed-capacity memory design, and hierarchical spatiotemporal organization, as well as for your constructive feedback on novelty and presentation. We have made the following revisions.

W1. Comprehensive Evaluation on HM3D.

We added a comprehensive baseline comparison on HM3D, a dataset of realistic indoor scans (revised Table 1). All models retain their corresponding MemoryMaze architectures and training schemes, with depth and hidden dimensions scaled to approximately 100M parameters (except CoME, which retains approximately 297M). Experiments were conducted over three random seeds.

GridWM achieves the lowest MSE and LPIPS among the original baselines under both protocols, reducing MSE/LPIPS by 15.7%/0.7% under 256→64 and 31.9%/23.3% under 500→500, respectively. Qualitative comparisons (Appendix Fig. 7) further demonstrate GridWM's ability to preserve scene structure and visual details over long rollouts. The HM3D configurations are available in our anonymous repository.

W2. Clarification of Novelty and Related Work.

We reorganized the Introduction to clarify that our central question is how to organize and utilize spatiotemporal information for long-horizon world modeling. Our two primary contributions are:

  • Hierarchical spatiotemporal organization: Systematic ablations show that temporal-to-spatial processing outperforms temporal-only, spatial-only, and joint encoding, with consistent benefits as Transformer depth increases (Fig. 3).
  • Content-conditioned spatial addressing: With only 0.05M additional parameters, learned per-head offsets improve prediction and exhibit meaningful geometric alignment with scene structures (Fig. 4, Appendix G).

Grid-like encoding serves as a biologically and theoretically motivated default. We have corrected the overly strong claims about its independent advantages and added robustness analyses (Appendix F). These revisions do not affect our two core contributions or their original experimental findings.

We also expanded the discussion of recent spatial memory methods. Detailed comparisons with ESWM and MemoryVLN are provided below.

W3. Missing References.

We added the original RoFormer citation for standard temporal RoPE and standardized the relevant terminology and notation.

Q. Comparison with ESWM and MemoryVLN.

The spatial memory mechanisms of ESWM and MemoryVLN differ fundamentally from GridWM.

ESWM infers spatial structure from episodic state–action transitions, while MemoryVLN integrates 6D poses with visual and geometric features for navigation memory retrieval. GridWM instead maintains path-integrated spatial addresses, organizes local temporal integration and long-range spatial retrieval in successive Transformer stages, and adapts retrieval addresses through content-conditioned offsets to support open-loop visual prediction.

We further implemented an adapted ESWM following the original paper and pseudocode (no public code can be found) and conducted controlled comparisons on MemoryMaze and HM3D (Appendix E.4, Table 7). GridWM achieves better perceptual prediction under limited historical coverage and remains competitive with a full-transition-memory model even in highly repetitive HM3D environments, despite using fixed-capacity memory. Revisit-stratified analysis further supports these findings.

Together, these additional experiments and analyses reinforce the value of GridWM's long-horizon prediction capability, hierarchical spatiotemporal organization, and spatial memory retrieval mechanisms.

Dear Reviewer,

Thank you for your time and thoughtful review. As the discussion period is approaching its end, we would greatly appreciate it if you could take a moment to review our rebuttal and additional experiments.

We would be happy to clarify any remaining concerns and look forward to your feedback.

Best regards, The Author

We thank you for recognizing GridWM’s architectural design and experimental results. We would like to further clarify its differences from related work and highlight additional evidence supporting our methodological contributions.

1. Comparison with MemoryVLN and Methodological Novelty.

Upon further examination, we realized that we had mistakenly confused the MemoryVLN paper by Wei et al. (2026) with the work by Hu et al. (2026) that you referenced. We apologize for this mistake.

Both methods employ bounded short- and long-term memories, but they differ in three key aspects:

(1) Spatiotemporal organization. MemoryVLN targets vision-language navigation, selects historical frames based on visual similarity, and organizes temporal, spatial, and trajectory information into separate memory modules. GridWM focuses on long-horizon open-loop visual prediction and investigates how local temporal integration and long-range spatial retrieval should be organized across Transformer layers. Fig. 3A–C independently validates this hierarchical design with offsets disabled. In particular, Fig. 3C shows that temporal layers benefit from contiguous history, while spatial layers benefit from long-term memory coverage.

(2) Utilization of spatial information. MemoryVLN provides object-level scene graphs and historical trajectory maps as additional context to the language model. GridWM directly incorporates path-integrated spatial relationships into historical attention through Spatial RoPE. Appendix E.3, Table 7 shows that directly injecting spatial representations into a Transformer, even with access to the full history, remains substantially less effective than GridWM (500→500 LPIPS: 0.411/0.418 vs. 0.292). This supports the importance of how spatial information is organized and utilized.

(3) Content-conditioned spatial addressing. MemoryVLN constructs spatial semantics through explicit object detection and scene graphs. GridWM introduces head-specific spatial address offsets with only 0.051M additional parameters, enabling content-adaptive memory addressing without explicit scene graphs. The new Appendix G, Fig. 12 reveals head-specific directional preferences and geometry-adaptive offsets. Targeted memory interventions (Fig. 12C–E) further show that different heads attend to historical observations of different walls, and perturbing their selected memories selectively affects the corresponding predicted regions. Intervening on both heads increases LPIPS from 0.114 to 0.175, providing direct evidence of historical memory utilization.

Together, these results further support our two specific methodological contributions: hierarchical spatiotemporal organization and content-conditioned spatial addressing.

2. Further Comparison with ESWM.

The new Appendix E.4, Table 8 and Fig. 9 compares GridWM with an adapted ESWM and analyzes their performance under different historical revisit frequencies. When directly matching historical viewpoints are unavailable, GridWM achieves lower LPIPS under both evaluation protocols. Conversely, under the highly repetitive HM3D 500→500 setting, ESWM benefits from retaining all observed state transitions and achieves better performance, while GridWM remains competitive with bounded memory. These findings reveal the complementary strengths of the two memory mechanisms under different levels of historical coverage.

3. Improvements in Presentation.

To address your concerns about readability, we systematically revised the Introduction and Section 2. We reorganized the methodological presentation, clarified component inputs, outputs, and data flow, simplified ambiguous terminology, moved the essential motion encoder details and loss definitions into the main text, and added the original RoPE citation. The complete implementation and configurations are also publicly available in our anonymous repository.

Thank you again for your thoughtful review. We would greatly appreciate your feedback at your earliest convenience and would be happy to discuss any remaining questions during the discussion period.

Could reviewers B3oH and 9P4y kindly provide some additional feedback, even if it is simply to let me know that my responses have not adequately addressed your concerns? I would greatly appreciate your feedback.

Comments on the paper

5 comments
AuthorsOpenReview ↗

Overall Response: Core Contributions and Major Revisions

We thank all three reviewers for their careful reviews and constructive feedback. We first clarify the paper's core contributions, then summarize its key strengths and revisions.

Core Contributions. HiSTFormer adopts biologically inspired multi-periodic grid-like encoding as its default spatial representation and investigates how to effectively organize and utilize spatiotemporal information for long-horizon visual prediction. Its main contributions are hierarchical temporal-to-spatial organization and content-conditioned spatial memory addressing, both supported by systematic ablations and mechanistic analyses.

Key Strengths (S: Strength; W: Weakness):

  • Long-horizon prediction: Robust extrapolation far beyond the training sequence length (all reviewers' summaries; B3oH Strength; 9P4y S1).
  • Bounded memory cost: Long-range memory access under a fixed memory budget (all reviewers' summaries; B3oH S2).
  • Hierarchical spatiotemporal organization: Systematic ablations demonstrate that temporal-to-spatial processing outperforms temporal-only, spatial-only, and joint spatiotemporal encoding (B3oH S3; 9P4y S2: interesting, intuitive, well-motivated; RFAN S1: insightful).
  • Depth scalability: Consistent performance gains with deeper hierarchical Transformer stacks (Fig. 3B).
  • Lightweight, interpretable content-conditioned offsets: Only 0.05M additional parameters enable meaningful spatial address adjustments (9P4y S2; Fig. 4; Appendix Fig. 12; further analysis motivated by RFAN W3).
  • Open-source implementation: Code has been anonymously available since the initial submission: https://anonymous.4open.science/r/GridWM-B30D/.

Major Revisions:

  1. Clarified research focus: Rewrote the Introduction to position grid-like encoding as a biologically and theoretically motivated default, moderate claims about its unique advantages, and emphasize hierarchical spatiotemporal organization and content-adaptive addressing as the core contributions. We acknowledge and apologize that the statements on spatial discriminability (L443/L448) exceeded the supporting evidence. These claims have been corrected, with further analysis provided in Appendix F. This correction does not affect our two core contributions or their existing experimental findings.
  2. Expanded experimental validation: Added comprehensive quantitative comparisons on HM3D (Table 1), qualitative results (Appendix Fig. 8), and a comparison with an adapted ESWM baseline (Appendix E.4).
  3. Strengthened mechanistic analyses: Added Appendix F to detail spatial encoding variants, analyze the long-range prediction benefits of multi-periodic encoding, and discuss the limitations of hexagonal geometry. Appendix G further examines head-specific directional preferences, their relationships with wall distances, and the effects of targeted memory interventions.
  4. Improved methodological clarity: Refined the organization of the Method section, standardized notation and data flow, and moved the complete training objective definitions into the main text. The updated repository also includes HM3D experimental configurations.

Red highlights indicate substantive revisions addressing key reviewer concerns or strengthening experimental evidence; blue highlights indicate wording improvements for clarity without changing the original meaning.

AuthorsOpenReview ↗

Update to Paper Title and Model Name

Dear Reviewers and Area Chair,

In response to the reviewers’ shared concerns regarding the emphasis on grid-like encoding, particularly the suggestion to reconsider the naming, we have renamed our paper from “GridWM: Grid-Like Spatial Indexing for Long-Horizon World Models” to “Hierarchical Spatiotemporal World Modeling with Content-Adaptive Spatial Addressing”.

The model is accordingly renamed HiSTFormer, and the abstract has been slightly revised to reflect this change.

We believe these changes more accurately reflect our two core contributions and clarify the paper’s technical focus.

We sincerely thank all reviewers for their constructive feedback.

Dear Reviewer 9P4y,

Apologies for the additional message. Following the reviewers' suggestions, we have renamed our paper from "GridWM: Grid-Like Spatial Indexing for Long-Horizon World Models" to "Hierarchical Spatiotemporal World Modeling with Content-Adaptive Spatial Addressing" to better reflect our core contributions.

May I kindly ask whether you have had a chance to review our latest revisions and yesterday's clarification? In particular, we would greatly appreciate knowing whether our explanation has addressed your remaining concerns about how hierarchical organization and content-conditioned addressing support long-term spatial memory retrieval.

We sincerely look forward to your feedback and would be happy to address any remaining questions.

Thank you for your time, and have a wonderful day!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.