acceptodds
Under review as a conference paper at ICLR 2027

Spatial World Model: Spatially Structured Latent Dynamics and Planning

Abstract

Latent world models have emerged as a compelling approach to learning environment dynamics through reward-free representation learning. However, their latent dynamics often remain entangled and geometrically unstructured, limiting their interpretability and utility for downstream planning. In this paper, we introduce Spatial World Model (SWM), a framework for learning spatially structured latent dynamics. SWM structures the latent space through two complementary geometric properties: orthogonality and directional ordering. Orthogonality enforces mutually orthogonal latent directions, while directional ordering arranges each latent-transition projection according to its corresponding state-transition component. These properties yield a spatially structured latent space that reflects the spatial organization of the environment. Building on this structure, we construct a goal-directed latent heuristic by projecting the current-to-goal latent displacement onto the learned directions. We incorporate this heuristic into Spatial-Aware Latent Planning (SLP), which guides search over a latent roadmap and composes local world-model predictions into long-horizon plans. Experiments on OGBench PointMaze validate the spatial structure learned by SWM and show that SLP outperforms comparison methods at larger goal offsets and longer execution horizons.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
world model, representation learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_NHu8OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes learning a more spatially structured world model (called SWM) by projecting latent transitions to learned orthogonal bases while aligning their ordering with the corresponding state transitions. It further proposes a SPATIAL-AWARE LATENT PLANNING (SLP), which constructs a roadmap of offline latent landmarks and searches it using a projected distance heuristic. Expriments show significant improvement on long-horizon planning for large mazes compared to prior JEPA WMs.

Critical strengths
  • The idea of having better spatially structured world models is well-motivated and reasonable.
  • The visualization and rank correlation support the claim on spatial structure (though requiring state information and being limited to mazes) and provide better interpretability
  • The paper performs well on mazes, especially for large mazes and long-horizon planning
Critical weaknesses

The major weakness is that it's unclear how this can be applied to more complicated non-maze environments.

  • It requires access to predefined task-relevant states and their transitions during training. For mazes, the agent’s position is an obvious choice. However, for robotic manipulation, the robot pose and gripper alone are not enough; the authors use cube positions, which are not given in a real-world setup and would require additional state estimation. Obtaining suitable supervision becomes increasingly difficult with multiple objects, deformation, occlusion and etc.
  • Given that the state information is available during training, a valuable baseline is building the graph directly from observed transitions, then run A star. The JEPA WMs could potentially offer better generalization, but the advantage is not shown.
  • Limitations on data with high intrinsic dimensionality: The authors admit that "landmark coverage and search cost of its roadmap grow rapidly with the dimensionality of the projected space" (L1004), and thus dont do SLP for cube (though it only increases dimensions from 2 to 4). Since SLP contributes substantially to long-horizon planning performance (shown in figure 6), this raises concerns about scalability and applicability to more complex environments. The authors also dont show any planning (e.g. CEM) for cube, so the only eval is limited to mazes.
Questions for authors

Please see weaknesses

  • Could you provide an A star baseline fror mazes?
  • Could you provide the CEM planning performance on cube, and potentially more robotic manipulation tasks?
  • Would be interesting to apply AdaJEPA's test-time adaptation to SWM
4 comments
AuthorsOpenReview ↗

Response to Reviewer NHu8 (Part 1)

We thank Reviewer NHu8 for recognizing the motivation for spatially structured world models, the interpretability of the learned structure, and the strong long-horizon planning results. We address the concerns below and provide additional experiments in the revised Appendix A.6.

1. Task-state supervision and applicability to manipulation.

We would like to clarify that SWM does not use cube positions or other object states as training supervision. Across OGBench Cube, Panda Lift, and Panda Reach, the spatial supervision contains only the three-dimensional end-effector transition, (Δx,Δy,Δz)(\Delta x,\Delta y,\Delta z). This information is naturally available on physical robots through joint encoders and forward kinematics. Object positions are neither provided during training nor used as planner inputs. The simulator uses object state only to evaluate task success.

The world model continues to receive visual observations and can therefore represent object interactions visually, while the end-effector transitions organize its latent spatial structure. We do not claim that end-effector supervision alone is sufficient for every manipulation problem. Tasks involving multiple objects, deformation, or severe occlusion may require richer task-state definitions or additional state estimation, which is an important direction for future work.

2. Ground-truth state-space A-star baseline.

We would like to clarify that SLP is itself an A∗A^*-like planner. It searches a latent roadmap using the accumulated roadmap cost g(i)g(i) and the learned spatial heuristic h(i)h(i). The distinction is that SLP performs this search entirely from visual observations and learned latent representations.

SLP also performs candidate-landmark retrieval in the learned projected transition space. Tables 1 and 5 in the paper show that projected retrieval substantially reduces per-query and total kNN time compared with retrieval in the full 192192-dimensional latent space, while preserving or improving planning success.

A graph constructed directly from ground-truth task states could provide a useful oracle upper bound. However, running A∗A^* with state-space distances would require access to the current and goal states during planning, or knowledge of their exact identities in the offline dataset. This information is not available to SWM or the comparison methods at test time and would bypass the learned world model.

If task states are available only during training, the planner must still associate the current and goal images with appropriate graph nodes. This visual-to-state association is precisely part of the representation and retrieval problem addressed by SWM. We therefore view ground-truth state-space A∗A^* as a privileged oracle rather than a directly comparable visual-planning baseline. We will clarify this distinction in the paper.

AuthorsOpenReview ↗

Response to Reviewer NHu8 (Part 2)

3. Higher-dimensional planning and requested CEM results.

We report additional planning experiments on OGBench Cube, RoboSuite Panda Lift, and RoboSuite Panda Reach in the revised Appendix A.6.

Table R1. Planning success rate (%) on three-dimensional manipulation tasks.

MethodOGBench Cube (50/25)Panda Lift (100/50)Panda Reach (100/50)
SWM+SLP (ours)——88.8 ± 1.8
SWM+CEM (ours)72.0 ± 5.125.5 ± 7.049.6 ± 12.8
LeWM70.0 ± 4.920.4 ± 5.514.0 ± 3.7

Results are mean ± standard deviation across five evaluation seeds (42–46). Column headers show execution budget/goal offset. OGBench Cube and Panda Reach use a 3 cm3 \ \mathrm{cm} success threshold, while Panda Lift uses 4 cm4 \ \mathrm{cm}. A dash indicates that SLP is not evaluated because the task requires timed gripper control.

SWM+CEM and LeWM use the same base architecture and CEM configuration. SWM+CEM achieves higher mean success than LeWM on all three tasks, including an improvement from 14.0% to 49.6% on Panda Reach. These same-planner comparisons show that SWM’s spatially structured representation remains beneficial without roadmap search.

We also evaluate SLP on Panda Reach, where the task requires three-dimensional end-effector motion without a timed grasp. SWM+SLP achieves 88.8% success, compared with 49.6% for SWM+CEM. This result extends SLP from two-dimensional maze navigation to a three-dimensional robotic environment.

We agree that the current landmark-based SLP does not establish scalability to arbitrary task dimensions. We do not apply it to OGBench Cube or Panda Lift because these tasks require deciding when to close the gripper. The current SLP heuristic represents spatial progress but does not model the timing of discrete interaction events. CEM is used for these tasks because it optimizes complete action sequences. Combining SWM’s spatial structure with temporal or mode-aware latent structure remains an important future direction.

4. AdaJEPA test-time adaptation.

We appreciate this insightful suggestion. AdaJEPA provides a strong mechanism for adapting latent dynamics at test time, while SWM provides explicit spatial organization through orthogonality and directional ordering. These properties are complementary: AdaJEPA could adapt the predictive dynamics online, while SWM could provide and preserve spatial structure during adaptation. Combining the two may enable spatially structured test-time adaptation, and we believe this is a promising direction for future work.

Thank you for the clarification and additional experiments! I misunderstood the states used in cube because the original submission only mentions “x, y, z” without mentioning that they're end-effector positions.

However, my concerns about the applicability of this method are not resolved. SWM+CEM shows only modest gains over LeWM on cube and lift. The strongest results rely on SLP, but unevaluated on grasping tasks. I notice that gripper is included in the original submission, but omitted here. As the authors mentioned,

We do not apply it to OGBench Cube or Panda Lift because these tasks require deciding when to close the gripper. The current SLP heuristic represents spatial progress but does not model the timing of discrete interaction events.

Does including gripper help address the stated limitation?

It seems that it's already quite hard to apply SLP to three-dimensional tasks, which makes me worried about its scalability and applicability to more complex tasks. The authors also agree that "Tasks involving multiple objects, deformation, or severe occlusion may require richer task-state definitions or additional state estimation."

Given these unaddressed concerns, I remain my original weak reject rating.

AuthorsOpenReview ↗

Clarification of the Roles and Scope of SWM and SLP

We thank Reviewer NHu8 for the follow-up. We would like to clarify two distinctions central to the intended scope of our work: the distinction between SWM and SLP, and the distinction between graph search and trajectory optimization.

SWM is our spatially structured world-model contribution, while SLP is an A∗A^*-like planning application introduced to demonstrate that SWM’s learned spatial structure provides a useful latent heuristic. We do not claim that SLP is a general-purpose manipulation planner or a replacement for action-sequence optimizers such as CEM or MPC. Its purpose is to demonstrate that the spatial structure learned by SWM can directly support long-horizon latent search.

SWM can organize the gripper state as another ordered latent direction, allowing the goal to specify a closed gripper. However, this does not determine when the gripper should close relative to reaching and contacting the object. The current SLP roadmap represents spatial progress but not contact-dependent event timing. Therefore, adding the gripper state alone does not address this limitation. Extending SLP with temporal or interaction-mode structure to handle contact-dependent event timing is an important direction for future work. This limitation concerns interaction timing rather than spatial dimensionality.

Finally, the planning gains do not arise solely from SLP. As shown in Figure 6 and Tables 3 and 4 in the main paper, SWM+CEM already outperforms the comparison methods without roadmap search or the SLP heuristic. Since SWM+CEM follows the same CEM configuration as LeWM, this comparison isolates the benefit of SWM’s spatially structured latent dynamics.

Reviewer_oEQCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

Spatial World Model structures the latent space through two complementary geometric properties: orthogonality and directional ordering. This paper constructs a goal-directed latent heuristic by projecting the current-to-goal latent displacement onto the learned directions and incorporates this heuristic into spatial-aware latent planning, which guides search over a latent roadmap and composes local world-model predictions into long-horizon plans.

Critical strengths
  1. This research approach is novel because it constructs a latent space by leveraging orthogonality and directional-ordering properties.
Critical weaknesses
  1. The environment is very simple; many methods were not specifically designed for it, which raises issues regarding fair comparison.
  2. Generalization is difficult. Planning scalability remains unvalidated beyond two-dimensional navigation. All task-level planning results concern PointMaze. Landmark coverage and search costs can increase rapidly with dimensionality.
Questions for authors
  1. Lack of experiments. Comparing SWM+SLP with SWM+CEM simultaneously introduces roadmap composition, graph search, and the heuristic. It is difficult to determine whether the performance improvement is actually attributable to SWM.
  2. How does the JEPA architecture achieve out-of-distribution (OOD) generalization?
  3. How does it generalize across different environments?
  4. How does it generalize to high-dimensional environments?
3 comments
AuthorsOpenReview ↗

Response to Reviewer oEQC (Part 1)

We thank Reviewer oEQC for recognizing the novelty of constructing latent dynamics through orthogonality and directional ordering. We address the concerns below and provide additional experiments in the revised Appendix A.6.

Weakness 1: Simplicity and fairness of PointMaze.

We understand the concern that PointMaze is visually simpler than robotic manipulation. However, its difficulty arises from long-horizon planning: Figure 6 and Tables 3–4 show that most comparison methods degrade substantially as the goal offset and execution horizon increase. All methods are evaluated using the same observations, datasets, sampled start–goal pairs, execution budgets, and evaluation seeds. SWM does not use the maze layout or any PointMaze-specific planning rule.

To evaluate SWM in more diverse environments, we add experiments on OGBench Cube and two RoboSuite tasks, Panda Lift and Panda Reach. For all three environments, SWM uses only three-dimensional end-effector transitions, (Δx,Δy,Δz)(\Delta x,\Delta y,\Delta z), as training supervision. Object positions are not provided as supervision or as planner inputs.

Table R1. Planning success rate (%) on three-dimensional manipulation tasks.

MethodOGBench Cube (50/25)Panda Lift (100/50)Panda Reach (100/50)
SWM+SLP (ours)——88.8 ± 1.8
SWM+CEM (ours)72.0 ± 5.125.5 ± 7.049.6 ± 12.8
LeWM70.0 ± 4.920.4 ± 5.514.0 ± 3.7

Results are mean ± standard deviation across five evaluation seeds (42–46). Column headers show execution budget/goal offset. OGBench Cube and Panda Reach use a 3 cm3 \ \mathrm{cm} success threshold, while Panda Lift uses 4 cm4 \ \mathrm{cm}. A dash indicates that SLP is not evaluated because the task requires timed gripper control.

Using the same CEM configuration, SWM achieves higher mean success than LeWM on all three tasks, including an improvement of 35.6 percentage points on Panda Reach. SWM+SLP further achieves 88.8% success on Panda Reach, demonstrating spatially aware planning in a three-dimensional robotic environment. These results show that SWM is not limited to PointMaze.

Weakness 2: Planning scalability.

We agree that landmark coverage and roadmap-search costs increase with the dimensionality of the projected space. This is a limitation of the current SLP roadmap planner rather than the SWM representation itself. The new Panda Reach experiment extends SLP from two-dimensional navigation to three-dimensional end-effector planning. However, we do not claim that the current roadmap implementation scales to arbitrary task dimensions.

SWM can also be combined with planners that do not require landmark search. SWM+CEM constructs no roadmap and is unaffected by its search cost. Its higher mean performance than LeWM under the same CEM configuration shows that the spatially structured world model remains beneficial independently of SLP. We use CEM for OGBench Cube and Panda Lift because these tasks additionally require deciding when to close the gripper, which is not represented by the current spatial-only SLP heuristic.

AuthorsOpenReview ↗

Response to Reviewer oEQC (Part 2)

Question 1: What improvement is attributable to SWM?

We clarify that two different controlled comparisons are provided:

  1. SWM+CEM versus LeWM: Both use the same base world-model architecture and CEM configuration. SWM+CEM introduces no roadmap, graph search, or SLP heuristic. Therefore, this comparison isolates the effect of SWM’s spatially structured latent construction. SWM+CEM outperforms LeWM across all nine PointMaze settings and achieves higher mean success on all three additional manipulation tasks.

  2. SWM+SLP versus SWM+CEM: Both use the same trained SWM. This comparison measures the benefit of SLP as an integrated planner, including roadmap composition, graph search, and the spatial heuristic. We agree that it does not isolate these three SLP components individually, and we will clarify this distinction.

Questions 2 and 3: OOD and cross-environment generalization.

We do not claim zero-shot OOD or cross-environment generalization in the current paper. In our experiments, we train and evaluate separate SWM and LeWM models for each dataset. Therefore, the PointMaze, OGBench Cube, Panda Lift, and Panda Reach results demonstrate that the same SWM construction can be applied across different environments, rather than that a single trained model transfers between them. Using the shared spatial meaning of the learned directions for cross-domain alignment is an interesting direction for future work.

Question 4: Higher-dimensional environments.

SWM is not restricted to two-dimensional spatial transitions. As reported in the revised Appendix A.6, the additional experiments organize three-dimensional end-effector transitions and evaluate downstream planning in OGBench Cube, Panda Lift, and Panda Reach. The Panda Reach result further demonstrates SLP in a three-dimensional roadmap. These experiments address scalability beyond two-dimensional navigation, while arbitrary-dimensional task-and-motion planning remains outside the current scope. Hierarchical or approximate landmark search and the integration of spatial and temporal interaction structure are promising future directions.

Thank you very much for the reviewer's response. However, as only some of the issues have been addressed, I will keep my score unchanged.

Reviewer_qqDzOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces the idea of learning how movements in the real world relate to changes in the high dimensional latent representations. Based on this idea, the paper proposes a planning method that uses the learned spatial information to guide long-term planning. Experiments on OGBench demonstrate that the proposed approach achieves better planning performance than baseline approaches, especially for longer planning tasks.

Critical strengths
  • The paper addresses an important problem of aligning learned latent representations with the spatial structure of the physical environment
  • The idea of using ranking instead of regression is interesting and well-motivated
Critical weaknesses

My main concerns are related to the evaluation:

  • Limited evaluation: The experiments focus mainly on PointMaze environments. Evaluating the proposed approach in more diverse and challenging scenarios would help demonstrate its effectiveness and generalizability.
  • Insufficient ablation study: A direct comparison between ranking-based and regression-based supervision would help better justify the proposed design choice.
Questions for authors

As discussed in the weaknesses section, a more comprehensive evaluation would significantly strengthen the evidence for the effectiveness and generalizability of the proposed approach, while also providing stronger justification for its design choices.

1 comment
AuthorsOpenReview ↗

Response to Reviewer qqDz

We thank Reviewer qqDz for recognizing the importance of aligning latent representations with the spatial structure of the physical environment and for the positive assessment of our ranking-based design.

1. Limited evaluation.

We agree that evaluation on more diverse environments strengthens the evidence for SWM. In the revised Appendix A.6, we add experiments on OGBench Cube and two RoboSuite tasks, Panda Lift and Panda Reach. For all three environments, SWM uses only three-dimensional end-effector transitions, (Δx,Δy,Δz)(\Delta x,\Delta y,\Delta z), as training supervision. Object positions are not provided as supervision or as inputs to the planner. SWM and LeWM are trained separately on each dataset, so these experiments evaluate applicability across environments rather than zero-shot transfer.

Table R1. Planning success rate (%) on three-dimensional manipulation tasks.

MethodOGBench Cube (50/25)Panda Lift (100/50)Panda Reach (100/50)
SWM+SLP (ours)——88.8 ± 1.8
SWM+CEM (ours)72.0 ± 5.125.5 ± 7.049.6 ± 12.8
LeWM70.0 ± 4.920.4 ± 5.514.0 ± 3.7

Results are mean ± standard deviation across five evaluation seeds (42–46). Column headers show execution budget/goal offset. OGBench Cube and Panda Reach use a 3 cm3 \ \mathrm{cm} success threshold, while Panda Lift uses 4 cm4 \ \mathrm{cm}.

SWM+CEM and LeWM use the same CEM configuration, allowing their comparison to isolate the effect of the learned world model. SWM+CEM achieves higher mean success than LeWM in all three environments, including an improvement of 35.6 percentage points on Panda Reach. On Panda Reach, SWM+SLP further improves success from 49.6% to 88.8%, showing that the learned spatial heuristic can support three-dimensional robotic planning. We do not apply SLP to Cube or Lift because these tasks require timed gripper control, which is outside the scope of the current spatial heuristic. PointMaze was selected for the main experiments because it provides a controlled setting for evaluating long-horizon planning, while these additional results demonstrate that SWM is not limited to two-dimensional maze navigation.

2. Ranking-based versus regression-based supervision.

We sincerely appreciate this question and would like to discuss it because it directly addresses the motivation for our ranking-based design. The direct answer is that regression-based supervision did not converge in our method and therefore did not produce a meaningful planning baseline.

One possible explanation is that latent coordinates have arbitrary numerical scales, while direct regression requires precise numerical alignment between physical transitions and latent values. Ranking instead constrains the relative relationships among observed transitions and remains invariant to monotonic rescaling.

The curse of dimensionality may further contribute to this difficulty (Bellman, 1961; Stone, 1982; Beyer et al., 1999; Aggarwal et al., 2001). For a simple illustration, dividing each dimension into 10 intervals produces 10310^3 regions in a three-dimensional space but 1019210^{192} regions in our latent space. With finite training data, the high-dimensional latent space is therefore extremely sparse. Precise numerical regression can be difficult under this sparsity, whereas ranking only compares relationships among observed samples.

This motivation is consistent with representation-learning methods that employ “soft” comparisons rather than precise numerical constraints in latent space. SimCLR (Chen et al., 2020) and MoCo (He et al., 2020) compare relative similarities between samples, while RankSim (Gong et al., 2022) and RnC (Zha et al., 2023) organize representations according to label rankings. Learning latent representations that numerically and linearly encode the physical world remains an interesting direction for future work.

We again thank the reviewer for this thoughtful question, which helped us clarify the motivation for our ranking-based design.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.