- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper deals with latent world models and their usage in MPC-based planning with CEM and gradient descent where the cost function is a difference in latent embeddings. The paper shows that shaping the latent space is necessary by explicitly linking it to privileged information on the ground truth distances in the problem space. They further show that when architectures are decomposed into visual encoders and adapters, training the (larger) visual encoder can be stopped early by detecting whether sufficient information (linked to privileged information on the state) is already present in the visual features.
Critical strengths
Comparing latent states directly is probably one of the most critical current weaknesses in the WM+MPC literature, so addressing the shape of the latent space is an important problem to work on.
Critical weaknesses
I would like to reassure the human co-authors of this paper that I did not use an LLM for my review and that thus their paper has not been uploaded to an LLM either. I see that as an advantage for the submitters, but it also comes with a downside: the paper needs to be understandable for a human.
NOVELTY
Shaping the latent space for planning is not a new idea and has been done in the past, even with references provided in this submission (eg. Wang et al 2026). The paper claims that these classical straightening approaches are insufficient and require task-distance supervision, but this comes with a high cost, which is unfortunately not discussed in the paper: the method actually requires access to the structure of the problem space, ie. ground truth distances between states. IMO this is an enormous drawback and it makes the method close to being handcrafted for this problem.
More generally speaking it is not clear to me what the high-level goal of this research is. As I see it, solving these planning problems (or, generally, science) can be addressed in multiple principal directions:
- with the goal of actually solving the (application related) goal at hand, eg. navigation, manipulation, etc. and therefore optimising the performance on a downstream task which should be as complex and realistic as possible.
- going deep into a certain methodology considered to have potential, eg. latent WMs and MPC, ignoring the fact that they are currently not competitive but trying to improve them. This can be particular interesting when this technology has more potential, has additional properties or fewer constraints than the current SoTA methods.
- providing theoretical insights into a problem.
The current work does neither of these.
Latent WMs combined with MPC planning (like CEM) are by far not competitive at the moment, the case in point being the fact that the WM literature works on highly simplified toy problems. The TwoRooms environment is very very far from what navigation can currently do. I do not argue against research into WMs and MPC, but this submission combines this research with very application oriented constraints as the work requires actual ground truth / privileged information on the task structure. Once this information is available (and in reality it is actually never), is the proposed framework still the most obvious choice?
For comparison on realistic work on navigation, a big body of work, less recent and up to recent, trains models on realistic outdoor scenarios, and deploys on real robots. For instance the two following methods, similar to this submission, estimate distances between latent states. However, as opposed to the submission, they do NOT need ground truth distances and perform estimates from trajectories only (temporal distances).
[a] Dhruv Shah, Benjamin Eysenbach, Gregory Kahn, Nicholas Rhinehart, Sergey Levine, Rapid Exploration for Open-World Navigation with Latent Goal Models, CORL 2021. [b] Dhruv Shah, Sergey Levine, ViKiNG: Vision-Based Kilometer-Scale Navigation with Geographic Hints, RSS 2022.
If information on shortest paths (eg. the task structure) is available, it can be directly used for supervision, again, enabling navigation in real world scenarios:
[c] Kiana Ehsani et al. Imitating Shortest Paths in Simulation Enables Effective Navigation and Manipulation in the Real World, CVPR 2024.
Similarly, realistic navigation agents can now be trained 100% in simulation on the HM3D dataset and they can be deployed as is on real robots to navigate in unseen environments. This paper shows that the task structure is learned be probing planning:
[d] Janny et al., Reasoning in visual navigation of end-to-end trained agents: a dynamical systems approach, CVPR 2025.
TECHNICAL SOUNDNESS
If we accept the premise of using privileged information in the WM+MPC framework, why would the proposed method be the most natural or the best choice? Given the ground truth structure of the problem space, isn't the task of aligning it with the latent space very closely linked to estimating the value function of the task, which many methods do, most prominently critics in RL, but also other lines of work? This line of work is also mentioned in the related work section, paragraph "learning latent geometry", but it is never mentioned why the proposed direct regression should be better. It seems to me that Bellman fitting should be more principled and I don't see its downsides.
The different formulations of latent space shaping have been compared for fixed weights for their corresponding losses. It is not clear where these weights come from and whether the conclusions hold for different weights.
Equation 4: the expressions in the "s.t." section are not constraints. These expressions are merely part of the computation graph of the expression to be optimized, and the formulation is misleading is it reads like an optimization under constraints. This is not a minor error and raises further doubts on the validity of the method.
Equation (5): there is quite a big literature on learning how to rank, a discussion would have been appreciated.
Same for Temporal Ordering: there is a quite a big literature on the arrow of time and representation learning by handling the order of time steps, eg. early work by VGG:
[e] Lyndsey C. Pickup, Zheng Pan, Donglai Wei, Yichang Shih, Changshui Zhang, Andrew Zisserman, Bernhard Schölkopf and William T. Freeman, Seeing the Arrow of Time, CVPR 2014.
The three losses s,geo,order seem to be ad-hoc and lack motivation and positioning wrt the SoTa on these known problems.
Detail: L237 the loss has changed from upper case to lower case
EXPERIMENTS AND RESULTS
As already said above, the experiments are in line with the very simplistic experiments of the WM literature. While I do understand that sometimes overly simplistic environments (actually, "toy problems") can be justified to specifically test some particular line of promising models if a breakthrough is expected later, I think the community should rethink its current view on world models. These kind of models are currently held in high esteem, which entirely contrasts with the toy problems they are capable of solving. However, at some point we need to have take-away messages from papers, knowledge gained, and I am not sure what the knowledge is we gained from these experiments.
The paper also does not compare the proposed method to any SoTA method. All the comparisons are ablations of the proposed method. This fact alone would lead me to recommend rejection from any top-level conference.
PRESENTATION
While the paper is not very hard to understand, it is written in a non-conventional and very linear way. Like many papers of the very very recent literature, details and even experimental results are distributed evenly across the paper, even put into the abstract, instead of setting an appropriate level of abstraction in each section. A scientific paper should make it's case first, start with a proper motivation.
Having numerical constants directly in the method section (eg. L139) raises questions on the origin of these constants.
SUMMARY AND CONCLUSION
The paper's findings can be resumed as exploiting privileged information on the problem's structure to improve latent WMs + MPC on problems on which other methods (eg. pure RL or IL) would have a field day. Exploiting privileged information can be extremely helpful, but it is application driven and normally employed when we want to optimize performance on a downstream task. It is unsure what it tells us for latent WMs, which are generally considered as the way to scale up learning on large amount of unlabelled data, in the wild.
The work does not compare to the SoTA, even on the single simple task it evaluates on.
Questions for authors
See "weaknesses" section.
8 comments
Response to Reviewer ztez (2/6)
More generally speaking it is not clear to me what the high-level goal of this research is. As I see it, solving these planning problems (or, generally, science) can be addressed in multiple principal directions:
with the goal of actually solving the (application related) goal at hand, eg. navigation, manipulation, etc. and therefore optimising the performance on a downstream task which should be as complex and realistic as possible.
going deep into a certain methodology considered to have potential, eg. latent WMs and MPC, ignoring the fact that they are currently not competitive but trying to improve them. This can be particular interesting when this technology has more potential, has additional properties or fewer constraints than the current SoTA methods.
providing theoretical insights into a problem.
The current work does neither of these.
We agree with this high-level classification of research/scientific contributions. Our goal is to be contributing along point 2) (improvements of latent WMs) and not 1) (improving real-world navigation solutions).
We believe that this combination (WMs and optimization in latent space) has great potential for many applications, for the following reasons.
- Using MPC in latent space inherits one of the main advantages of MPC over policy learning: New behaviors can be generated at deployment time simply be changing the objective of the MPC problem, there is no need to train a new policy for every new task.
- While it is currently only used for planning in our paper, optimization in latent space has been shown to have potential for other tasks, for example latent safety filters
- Even if policy learning is the method of choice, there is existing evidence that model-based MPC, for example in the form of a learned dynamics model in latent space, is more sample-efficient than model-free reinforcement learning.
Therefore, we argue that investigation of latent world models like JEPA is of value even if currently domain and task-specific solutions outperform it. In particular, our paper
- shows and quantifies an important problem of latent space geometry when used for planning
- provides a methodology to fix these latent spaces that is fairly general, computationally cheap, and does not require extensive engineering effort (unlike, e.g., training value functions that can adapt or compensate for the unfavorable geometry of the latent space).
Latent WMs combined with MPC planning (like CEM) are by far not competitive at the moment, the case in point being the fact that the WM literature works on highly simplified toy problems. The TwoRooms environment is very very far from what navigation can currently do. I do not argue against research into WMs and MPC, but this submission combines this research with very application oriented constraints as the work requires actual ground truth / privileged information on the task structure. Once this information is available (and in reality it is actually never), is the proposed framework still the most obvious choice?
- As argued in the previous reply, the regularization with does not require strong priviledged information, and its applicability is not limited to simple planar navigation, as the new results in the Cube benchmark show.
- That the proposed framework is the most obvious choice is not our claim. However, we do want to repeat the point from above that learning a latent space with metrically consistent structure is of broader interest and relevance, even if a dedicated task-based methods may outperform this particular method in terms of planning preformance.
Response to Reviewer ztez (3/6)
For comparison on realistic work on navigation, a big body of work, less recent and up to recent, trains models on realistic outdoor scenarios, and deploys on real robots. For instance the two following methods, similar to this submission, estimate distances between latent states. However, as opposed to the submission, they do NOT need ground truth distances and perform estimates from trajectories only (temporal distances).
[a] Dhruv Shah, Benjamin Eysenbach, Gregory Kahn, Nicholas Rhinehart, Sergey Levine, Rapid Exploration for Open-World Navigation with Latent Goal Models, CORL 2021. [b] Dhruv Shah, Sergey Levine, ViKiNG: Vision-Based Kilometer-Scale Navigation with Geographic Hints, RSS 2022.
If information on shortest paths (eg. the task structure) is available, it can be directly used for supervision, again, enabling navigation in real world scenarios:
[c] Kiana Ehsani et al. Imitating Shortest Paths in Simulation Enables Effective Navigation and Manipulation in the Real World, CVPR 2024.
Similarly, realistic navigation agents can now be trained 100% in simulation on the HM3D dataset and they can be deployed as is on real robots to navigate in unseen environments. This paper shows that the task structure is learned be probing planning:
[d] Janny et al., Reasoning in visual navigation of end-to-end trained agents: a dynamical systems approach, CVPR 2025.
- We agree that these papers show impressive real-world performance of navigation solutions, but our point is not to focus on navigation per se. The proposed methodology from the papers does not transfer to, for example, robotic manipulation experiments, while WM+MPC promises this kind of generalization. We acknowledge that WM+MPC does currently have limitations, such as the latent geometry, which is exactly what we are aiming to show and improve on in this submission. We will make this clearer in our paper.
TECHNICAL SOUNDNESS
If we accept the premise of using privileged information in the WM+MPC framework, why would the proposed method be the most natural or the best choice? Given the ground truth structure of the problem space, isn't the task of aligning it with the latent space very closely linked to estimating the value function of the task, which many methods do, most prominently critics in RL, but also other lines of work? This line of work is also mentioned in the related work section, paragraph "learning latent geometry", but it is never mentioned why the proposed direct regression should be better. It seems to me that Bellman fitting should be more principled and I don't see its downsides.
- Yes, learning a critic, at least for goal-conditioned reinforcement learning, is directly related to learning the cost-to-go.
However, in our setting, because we assume that the ground-truth distance is known, we regress latent distances directly onto it, while Bellman fitting estimates the cost-to-go by bootstrapping from the current estimate. Bootstrapping propagates errors from the current estimate and adds hyperparameters, and Destrade et al. (2026), who learn such a critic, report training instabilities and inaccurate value functions. We argue that direct regression is the natural choice when privileged information is available.
However, the advantage of the proposed method is that it comes at a lower computational cost and engineering effort. Learning a reliable critic comes with its own challenges, as Destrade et al. (2026), who propose to do exactly that, note. By simply using the temporal ordering, such training difficulties and costs can be avoided.
- Furthermore, learning a good latent geometry is interesting because of other optimization-based tasks that this may unlock in latent space, although the validation of this thesis remains for future work.
Response to Reviewer ztez (1/6)
We thank the reviewer for the careful and detailed review of the paper. We address all raised points in the comments below. To avoid repetition we kindly ask the reviewer to find the details of added experiments in the official comment at the top of the page.
We appreciate the transparency, and likewise assure the reviewer that our paper and responses are human-generated.
We agree that adding ground-truth distances to training would require having significantly more privileged knowledge than the temporal straightening technique of Wang et al. 2026. This was precisely our motivation to also propose the supervision technique with Lorder, which requires no such information. However, we acknowledge that we may not have made this point clear enough, and that by training the adapter method on a mix of suboptimal and optimal (A*-generated) plans may have colluded effects of data vs. method, which reviewer tyNZ also pointed out. We are thus suggesting the following changes:
As described in our response to Reviewer tyNZ (W1, Table 2), we have added a new experiment that disentangles the effect of the optimality of the dataset from the latent reshaping. Interestingly, we found that temporal straightening by Wang et al. 2026 benefits strongly from optimal datasets (from 43.3% to 74.0%), although our method still outperforms it there (81.9%). However, the main advantage of our method with Lorder is that it has much weaker requirements on the optimality of the provided plans, so it also works on the suboptimal dataset (88.7%), where temporal straightening does significantly worse (43.3%).
We hope that the addition of the new benchmark, as described in our response to Reviewer HUKx (Q1; +21.4 points on Cube), also underlines that the method is not merely handcrafted for this particular use case. We do acknowledge that there are limitations to the approach, and suggest to add the following in the Discussion section of the paper:
The second technical contribution of the paper, the sufficiency test, does require ground-truth state information. We acknowledge that this privileged information may not always be available (but quite often it is, see remark below), so this part of the contribution does not always apply. We propose the following addition in the limitations section:
While we hope that these points address the reviewer's concerns, we do want to point out that we believe that privileged information in the form of optimal distances, optimal plans, etc., is not uncommonly avialable when distilling world models from simulators. For robotics problems, this is a valuable usecase as it still allows for creating fast and reactive policies at runtime, with privileged information only used at training time.