- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper proposes learning a more spatially structured world model (called SWM) by projecting latent transitions to learned orthogonal bases while aligning their ordering with the corresponding state transitions. It further proposes a SPATIAL-AWARE LATENT PLANNING (SLP), which constructs a roadmap of offline latent landmarks and searches it using a projected distance heuristic. Expriments show significant improvement on long-horizon planning for large mazes compared to prior JEPA WMs.
Critical strengths
- The idea of having better spatially structured world models is well-motivated and reasonable.
- The visualization and rank correlation support the claim on spatial structure (though requiring state information and being limited to mazes) and provide better interpretability
- The paper performs well on mazes, especially for large mazes and long-horizon planning
Critical weaknesses
The major weakness is that it's unclear how this can be applied to more complicated non-maze environments.
- It requires access to predefined task-relevant states and their transitions during training. For mazes, the agent’s position is an obvious choice. However, for robotic manipulation, the robot pose and gripper alone are not enough; the authors use cube positions, which are not given in a real-world setup and would require additional state estimation. Obtaining suitable supervision becomes increasingly difficult with multiple objects, deformation, occlusion and etc.
- Given that the state information is available during training, a valuable baseline is building the graph directly from observed transitions, then run A star. The JEPA WMs could potentially offer better generalization, but the advantage is not shown.
- Limitations on data with high intrinsic dimensionality: The authors admit that "landmark coverage and search cost of its roadmap grow rapidly with the dimensionality of the projected space" (L1004), and thus dont do SLP for cube (though it only increases dimensions from 2 to 4). Since SLP contributes substantially to long-horizon planning performance (shown in figure 6), this raises concerns about scalability and applicability to more complex environments. The authors also dont show any planning (e.g. CEM) for cube, so the only eval is limited to mazes.
Questions for authors
Please see weaknesses
- Could you provide an A star baseline fror mazes?
- Could you provide the CEM planning performance on cube, and potentially more robotic manipulation tasks?
- Would be interesting to apply AdaJEPA's test-time adaptation to SWM
4 comments
Response to Reviewer NHu8 (Part 2)
3. Higher-dimensional planning and requested CEM results.
We report additional planning experiments on OGBench Cube, RoboSuite Panda Lift, and RoboSuite Panda Reach in the revised Appendix A.6.
Table R1. Planning success rate (%) on three-dimensional manipulation tasks.
| Method | OGBench Cube (50/25) | Panda Lift (100/50) | Panda Reach (100/50) |
|---|---|---|---|
| SWM+SLP (ours) | — | — | 88.8 ± 1.8 |
| SWM+CEM (ours) | 72.0 ± 5.1 | 25.5 ± 7.0 | 49.6 ± 12.8 |
| LeWM | 70.0 ± 4.9 | 20.4 ± 5.5 | 14.0 ± 3.7 |
Results are mean ± standard deviation across five evaluation seeds (42–46). Column headers show execution budget/goal offset. OGBench Cube and Panda Reach use a success threshold, while Panda Lift uses . A dash indicates that SLP is not evaluated because the task requires timed gripper control.
SWM+CEM and LeWM use the same base architecture and CEM configuration. SWM+CEM achieves higher mean success than LeWM on all three tasks, including an improvement from 14.0% to 49.6% on Panda Reach. These same-planner comparisons show that SWM’s spatially structured representation remains beneficial without roadmap search.
We also evaluate SLP on Panda Reach, where the task requires three-dimensional end-effector motion without a timed grasp. SWM+SLP achieves 88.8% success, compared with 49.6% for SWM+CEM. This result extends SLP from two-dimensional maze navigation to a three-dimensional robotic environment.
We agree that the current landmark-based SLP does not establish scalability to arbitrary task dimensions. We do not apply it to OGBench Cube or Panda Lift because these tasks require deciding when to close the gripper. The current SLP heuristic represents spatial progress but does not model the timing of discrete interaction events. CEM is used for these tasks because it optimizes complete action sequences. Combining SWM’s spatial structure with temporal or mode-aware latent structure remains an important future direction.
4. AdaJEPA test-time adaptation.
We appreciate this insightful suggestion. AdaJEPA provides a strong mechanism for adapting latent dynamics at test time, while SWM provides explicit spatial organization through orthogonality and directional ordering. These properties are complementary: AdaJEPA could adapt the predictive dynamics online, while SWM could provide and preserve spatial structure during adaptation. Combining the two may enable spatially structured test-time adaptation, and we believe this is a promising direction for future work.
Thank you for the clarification and additional experiments! I misunderstood the states used in cube because the original submission only mentions “x, y, z” without mentioning that they're end-effector positions.
However, my concerns about the applicability of this method are not resolved. SWM+CEM shows only modest gains over LeWM on cube and lift. The strongest results rely on SLP, but unevaluated on grasping tasks. I notice that gripper is included in the original submission, but omitted here. As the authors mentioned,
We do not apply it to OGBench Cube or Panda Lift because these tasks require deciding when to close the gripper. The current SLP heuristic represents spatial progress but does not model the timing of discrete interaction events.
Does including gripper help address the stated limitation?
It seems that it's already quite hard to apply SLP to three-dimensional tasks, which makes me worried about its scalability and applicability to more complex tasks. The authors also agree that "Tasks involving multiple objects, deformation, or severe occlusion may require richer task-state definitions or additional state estimation."
Given these unaddressed concerns, I remain my original weak reject rating.
Clarification of the Roles and Scope of SWM and SLP
We thank Reviewer NHu8 for the follow-up. We would like to clarify two distinctions central to the intended scope of our work: the distinction between SWM and SLP, and the distinction between graph search and trajectory optimization.
SWM is our spatially structured world-model contribution, while SLP is an -like planning application introduced to demonstrate that SWM’s learned spatial structure provides a useful latent heuristic. We do not claim that SLP is a general-purpose manipulation planner or a replacement for action-sequence optimizers such as CEM or MPC. Its purpose is to demonstrate that the spatial structure learned by SWM can directly support long-horizon latent search.
SWM can organize the gripper state as another ordered latent direction, allowing the goal to specify a closed gripper. However, this does not determine when the gripper should close relative to reaching and contacting the object. The current SLP roadmap represents spatial progress but not contact-dependent event timing. Therefore, adding the gripper state alone does not address this limitation. Extending SLP with temporal or interaction-mode structure to handle contact-dependent event timing is an important direction for future work. This limitation concerns interaction timing rather than spatial dimensionality.
Finally, the planning gains do not arise solely from SLP. As shown in Figure 6 and Tables 3 and 4 in the main paper, SWM+CEM already outperforms the comparison methods without roadmap search or the SLP heuristic. Since SWM+CEM follows the same CEM configuration as LeWM, this comparison isolates the benefit of SWM’s spatially structured latent dynamics.
Response to Reviewer NHu8 (Part 1)
We thank Reviewer NHu8 for recognizing the motivation for spatially structured world models, the interpretability of the learned structure, and the strong long-horizon planning results. We address the concerns below and provide additional experiments in the revised Appendix A.6.
1. Task-state supervision and applicability to manipulation.
We would like to clarify that SWM does not use cube positions or other object states as training supervision. Across OGBench Cube, Panda Lift, and Panda Reach, the spatial supervision contains only the three-dimensional end-effector transition, (Δx,Δy,Δz). This information is naturally available on physical robots through joint encoders and forward kinematics. Object positions are neither provided during training nor used as planner inputs. The simulator uses object state only to evaluate task success.
The world model continues to receive visual observations and can therefore represent object interactions visually, while the end-effector transitions organize its latent spatial structure. We do not claim that end-effector supervision alone is sufficient for every manipulation problem. Tasks involving multiple objects, deformation, or severe occlusion may require richer task-state definitions or additional state estimation, which is an important direction for future work.
2. Ground-truth state-space A-star baseline.
We would like to clarify that SLP is itself an A∗-like planner. It searches a latent roadmap using the accumulated roadmap cost g(i) and the learned spatial heuristic h(i). The distinction is that SLP performs this search entirely from visual observations and learned latent representations.
SLP also performs candidate-landmark retrieval in the learned projected transition space. Tables 1 and 5 in the paper show that projected retrieval substantially reduces per-query and total kNN time compared with retrieval in the full 192-dimensional latent space, while preserving or improving planning success.
A graph constructed directly from ground-truth task states could provide a useful oracle upper bound. However, running A∗ with state-space distances would require access to the current and goal states during planning, or knowledge of their exact identities in the offline dataset. This information is not available to SWM or the comparison methods at test time and would bypass the learned world model.
If task states are available only during training, the planner must still associate the current and goal images with appropriate graph nodes. This visual-to-state association is precisely part of the representation and retrieval problem addressed by SWM. We therefore view ground-truth state-space A∗ as a privileged oracle rather than a directly comparable visual-planning baseline. We will clarify this distinction in the paper.