acceptodds
Under review as a conference paper at ICLR 2027

Cognitive Map World Model

Abstract

World models have emerged as a promising paradigm for spatial intelligence, yet current approaches typically treat action-observation pairs as unstructured sequences, neglecting the intrinsic algebraic properties of spatial actions and leading to data inefficiency. We propose the Cognitive Map World Model (CMWM), which integrates the Abelian group structure of spatial actions directly into the latent representation. Inspired by grid cell phase coding, CMWM maps continuous actions to phase shifts across multi-scale periodic modules, constructing a globally cognitive map that eliminates recursive prediction errors and ensures stable spatial memory and imagination. Furthermore, this structured latent space redefines navigation from expensive iterative planning to a congruence solving problem, enabling constant-time trajectory generation. Experiments demonstrate that CMWM significantly outperforms autoregressive and MaskGIT baselines in prediction fidelity, convergence speed, and navigation efficiency.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Brain-inspired AI, World models, Cognitive map, Phase coding

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_qaL7OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes the Cognitive Map World Model (CMWM), which explicitly incorporates the Abelian group structure of spatial actions into a world model. Inspired by grid-cell phase coding, the model encodes spatial actions as phase shifts across multiple periodic modules and aligns the latent representation of observations with these phase shifts. This enables direct prediction of future observations without autoregressive generation. To support navigation, the authors propose estimating relative displacement between two observations by solving a congruence problem. Experiments show improvement over the autoregressive and MaskGiT baselines.

Critical strengths
  1. The proposed framework is mathematically well-motivated, and the closed-form latent alignment solution is particularly compelling.

  2. CMWM avoids autoregressive generation using an explicit, structured latent dynamic. This is an insightful idea that could improve computational efficiency substantially

Critical weaknesses
  1. Does multi-periodic phase encoding really have empirical advantages? Although App 4 includes several ablations, they all assume multi-periodic phase encoding and modify the design on top of it. In particular, the core computational advantage of directly predicting distant future states from accumulated actions does not inherently require multi-periodic encoding. Therefore, I wonder whether multi-periodic phase encoding has empirical benefit over a non-periodic additive latent representation (which could have the same computational advantage).
Questions for authors

None

2 comments

Response to Reviewer qaL7

We thank the reviewer for the positive assessment of the mathematical formulation and closed-form alignment. Your question identifies an important distinction: horizon-independent state prediction follows from explicitly composable dynamics; it does not inherently require periodic encoding. Our claim should therefore concern the empirical usefulness of the representation, not exclusive ownership of the computational advantage.

New representation ablations

We evaluated two additional variants of the phase-based model: a single-period representation, and eight modules with all translation periods enlarged by 14,000×14{,}000\times. Both retain direct prediction from accumulated actions. We compare their results with the default configuration reported in Table 4:

RepresentationTrain PSNR ↑Test PSNR ↑
CMWM: eight original-scale periods (Table 4)22.17 dB17.61 dB
Single-period variant10.56 dB10.60 dB
Eight large-period modules: translation periods ×14,000\times 14{,}00011.35 dB11.32 dB

The default representation exceeds the single-period and large-period variants by 7.01 dB and 6.29 dB in test PSNR, respectively. Because direct prediction is retained, these results indicate that its computational advantage alone is insufficient to explain the default model’s prediction quality. The eight-module control further shows that the number of periods alone does not explain the improvement: their spatial scales matter.

A plausible explanation is the balance between spatial sensitivity and range. A single period can introduce position ambiguity over repeated cycles, whereas greatly enlarged periods reduce local phase changes: Eq. (1) gives a 14,000-fold smaller phase increment for the same translation. Multi-scale coding combines locally sensitive modules with longer-range information. The results are consistent with this explanation, but do not independently establish its causal mechanisms.

Scope of the conclusion

The large-period variant probes a low-frequency phase representation; it is not equivalent to an unconstrained non-periodic additive latent model. We therefore do not claim that these experiments establish universal superiority over additive representations. They provide evidence for the original multi-scale configuration within CMWM, while a matched additive-latent baseline remains an important comparison.

We will add these results and explicitly distinguish the benefits of composable dynamics for direct prediction from those of multi-scale phase coding for representation quality.

Dear Reviewers,

The authors have addressed your comments in their rebuttal. Please review their responses and provide any further feedback or follow-up questions, if any. Thank you!

Best, AC

Reviewer_TppEOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

Paper proposes a novel world model - Cognitive Map World Model (CMWM). CMWM follows the encoder-decoder framework the observation sequence features are encoded in the latent space and aligned with the action sequence. The novelty lies in the alignment which actions are transformed into phase shifts across multiple periodic modules. The aligned latent representation is then passed to a decoder for reconstruction of the original observation. Paper claims that the latent transformation is inspired by neural science in band cells and grid cells encoding of spatial information.

The proposed navigation framework poses navigation as a congruence problem in phase space, significantly reducing computational complexity compared to the conventional model predictive control problem posing approach.

Empirical experiments utilized the robotics datasets SCAND, TartanDrive, HuRoN and RECON for training and evaluation. For spatial memory and spatial imagination tasks, LPIPS and PSNR metric were used. For synthesis quality assessment, FID was used. Baseline methods are AR and MaskGIT.

The most prominent advantage of CMWM lies in its computational efficiency, as shown in complexity analysis of the comparative methods, CMWM has a complexity of constant time with respect to planning horizon T and optimization iterations, N_CEM.

Critical strengths
  1. Paper proposes a novel world model which is computationally efficient for solving the navigation problem.
  2. Experiments were done for 4 robotic datasets with 2 other baseline methods.
Critical weaknesses
  1. Only 2 baseline methods were compared against. This brings into question the coverage and completeness of the empirical experiment design.
  2. The metrics were based on predictive image quality and not navigation task completeness metric such as Success Rate(SR) and Success weighted by Path Length(SPL) metric. Hence, its value proposition as a navigation memory/prediction module is not well aligned using the LPIPS, PSNR and FID. This is especially the case when the comparative methods are designed for navigation, i.e. a trajectory optimisation problem, rather than a image generation / next-step prediction problem.
Questions for authors

Refer to Point 2 in Weaknesses about the comparison metrics.

2 comments

Response to Reviewer TppE

We thank the reviewer for recognizing the structured world-model formulation and its computational potential. We agree that baseline coverage and the distinction between localization and complete navigation require clarification.

1. Baseline coverage and Navigation World Models

Our AR and MaskGIT baselines use common ViT building blocks, the same image tokenizer and inputs, and the same 100-epoch training schedule. This design helps compare modeling approaches under controlled conditions, but does not establish superiority over all representative navigation world models.

In particular, Navigation World Models (NWM) is an important related method. The available NWM results use a different train/test protocol from ours, so comparing published scores directly would confound model differences with evaluation differences. We acknowledge the absence of a matched NWM evaluation rather than treating our AR baseline as an NWM reproduction.

CMWM’s contribution is a structured representation supporting both observation prediction and direct relative displacement estimation, rather than a claim of superior image realism. Its computational distinction is avoiding repeated rollout optimization for the start–goal displacement query. We will not extrapolate this distinction into an unsupported claim that NWM cannot handle long-distance navigation.

2. Evaluation metrics: localization versus navigation success

We agree that PSNR, LPIPS, and FID cannot establish navigation success. In our experiments, these metrics evaluate observation prediction, while Table 7 separately measures relative localization error:

Test-set localization error ↓CMWMMaskGITAR
Short range0.45±0.10 m0.45 \pm 0.10\,\mathrm{m}0.89±0.47 m0.89 \pm 0.47\,\mathrm{m}1.20±0.68 m1.20 \pm 0.68\,\mathrm{m}
Long range0.48±0.28 m0.48 \pm 0.28\,\mathrm{m}Not reportedNot reported

Thus, the spatial utility of the representation is evaluated beyond image quality. Nevertheless, these are offline displacement estimates, and Figure 6E illustrates geometric trajectory construction rather than collision-free execution. We will relabel and describe this evaluation accordingly.

SR and SPL are appropriate for a complete, executed navigation system, but not directly applicable to an isolated displacement estimator. Our current results support CMWM as a localization and spatial-prediction module that could guide a local planner; they do not establish end-to-end navigation performance. We will explicitly narrow the navigation claims and identify closed-loop SR/SPL evaluation as an outstanding requirement.

Dear Reviewers,

The authors have addressed your comments in their rebuttal. Please review their responses and provide any further feedback or follow-up questions, if any. Thank you!

Best, AC

Reviewer_bupxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces the Cognitive Map World Model (CMWM), an encoder–decoder framework that incorporates an Abelian structure of spatial displacements into its latent representation. Continuous displacements are mapped to phase shifts across multiple periodic modules, and a closed-form alignment procedure enforces consistency between observation-derived phases and action-induced transitions. Future observations are generated by propagating the aligned phases using accumulated actions, without recursively feeding back predicted observations. The learned representation also supports relative displacement estimation between start and goal observations through a congruence-solving procedure. Experiments on robot exploration datasets compare CMWM with autoregressive and MaskGIT-style baselines, reporting improved long-horizon prediction quality and relative localization accuracy, alongside architectural ablations and a joint multi-map learning experiment.

Critical strengths

1.The method has a clear motivation for structured modeling. It explicitly incorporates spatial action consistency into the latent representation and uses the same representation for observation prediction and relative localization, resulting in a coherent overall design. 2.The phase alignment procedure has a well-defined optimization objective and a closed-form solution, with a derivation provided in the appendix. This makes the core alignment mechanism interpretable and verifiable. 3.The advantage in long-horizon prediction is supported by both quantitative and qualitative results. Figure 4 and Tables 5–6 show that CMWM maintains relatively stable prediction quality as the prediction horizon increases, with less degradation than the evaluated AR and MaskGIT baselines.

Critical weaknesses

1.Some claims lack direct experimental support. The paper does not provide training curves or comparisons of the computational resources required to reach comparable performance, making it difficult to substantiate the claims of greater training stability and faster convergence. The efficiency analysis primarily compares the number of forward passes and lacks end-to-end runtime measurements. Figure 4A supports performance improvements within the evaluated model sizes, but provides limited evidence for broader scalability. 2.The contributions of the core design components have not been sufficiently isolated. Existing ablations mainly examine configurations within the phase-based framework, without adequately separating the benefits of multi-scale phase representations, action-consistent alignment, and non-autoregressive prediction. Consequently, the overall performance differences relative to AR and MaskGIT are insufficient to establish the contribution of the phase representation itself. 3.The representativeness of the baselines remains limited. The experiments primarily use AR-style and MaskGIT-style baselines constructed within a common ViT architecture, which is insufficient to comprehensively assess the proposed method against representative world models. In particular, direct comparisons with methods closely related in task and dataset settings, such as Navigation World Models [1], are missing. 4.Error correction during practical navigation is insufficiently discussed and evaluated. The paper does not clearly explain how new observations collected during execution are used to update the state, or how action measurement noise and control errors affect phase propagation and displacement recovery. The current offline relative localization results and geometric trajectory demonstrations provide limited support for closed-loop navigation capabilities and effectiveness across a broad range of downstream tasks. Minor comment: The writing of the paper could be improved. [1] Bar, A., Zhou, G., Tran, D., Darrell, T., & LeCun, Y. (2025). Navigation World Models. CVPR.

Questions for authors

see the weakness

2 comments

Response to Reviewer bupx

We thank the reviewer for recognizing the coherent design, interpretable alignment procedure, and long-horizon prediction results. We address the four concerns below.

1. Evidence for convergence, scaling, and efficiency

We agree that these claims should be separated. Appendix C reports the same 100-epoch training schedule for all models, but this does not establish faster convergence or greater training stability. We will remove those unsupported claims. Figure 4A demonstrates improvements only within the evaluated model sizes; we will describe this as a model-size study rather than evidence of general scaling laws.

Table 1 measures network forward-pass counts, not end-to-end latency. CMWM avoids horizon-dependent recursive prediction and repeated rollout optimization, but encoding, decoding, and congruence solving still incur computation. We will qualify “constant time” accordingly and remove unmeasured orders-of-magnitude runtime claims.

2. Isolating the benefit of multi-scale phase representations

We agree that direct prediction alone does not establish the value of periodic coding. We additionally evaluated a single-period variant and an eight-module variant with all translation periods enlarged by 14,000×14{,}000\times. The reference is the default configuration in Table 4:

RepresentationTrain PSNR ↑Test PSNR ↑
CMWM: eight original-scale periods (Table 4)22.17 dB17.61 dB
Single-period variant10.56 dB10.60 dB
Eight large-period modules: translation periods ×14,000\times 14{,}00011.35 dB11.32 dB

The reported test gaps are 7.01 dB and 6.29 dB, respectively. Both variants retain the direct-prediction formulation, so these results support the importance of the period configuration beyond simply avoiding autoregression. In particular, the eight-module control shows that retaining multiple modules alone is insufficient when their spatial scales are greatly enlarged. From Eq. (1), enlarging a translation period by 14,000×14{,}000\times reduces its phase response to a given displacement by the same factor, suggesting reduced local spatial sensitivity.

These controls strengthen the representation analysis, but do not separately quantify alignment’s contribution or substitute for a fully non-periodic additive-latent baseline. We will make this scope explicit.

3. Baseline coverage and Navigation World Models

We agree that Navigation World Models (NWM) is an important comparison. Our shared-ViT AR and MaskGIT implementations provide controlled architectural comparisons, not a comprehensive benchmark against published systems. The available NWM results use a different train/test protocol from ours, so directly juxtaposing their reported numbers would not be a matched comparison. We acknowledge that this does not replace evaluating NWM under a common protocol.

Our intended distinction is direct start–goal displacement recovery without optimizing an intervening rollout, not a claim that NWM cannot navigate long distances. We do not claim superiority over NWM in image synthesis or closed-loop navigation, and will clarify this positioning.

4. Online state updates and execution errors

A natural online use of CMWM is to: (i) encode the latest real observation; (ii) recover its displacement to the goal from their phase differences; (iii) pass this estimate to a local controller; and (iv) execute a short segment and repeat. A recent window of real observations and measured displacements can also be re-aligned using Eq. (7). Thus, the state need not be propagated indefinitely from the initial observation alone.

However, re-observation is not itself evidence of robustness to noisy actions, and Eq. (7) assumes accurate displacement constraints. Table 7 reports offline relative localization—test errors of 0.45±0.10 m0.45 \pm 0.10\,\mathrm{m} at short range and 0.48±0.28 m0.48 \pm 0.28\,\mathrm{m} at long range—not closed-loop navigation success. We will distinguish this demonstrated capability from the proposed online integration and narrow the downstream-task claims accordingly.

Dear Reviewers,

The authors have addressed your comments in their rebuttal. Please review their responses and provide any further feedback or follow-up questions, if any. Thank you!

Best, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.