acceptodds
Under review as a conference paper at ICLR 2027

From Representation Learning to Reshaping Latent Geometry in World Models

Abstract

Latent world models encode high-dimensional observations into compact representations, where the dynamics can be learned and used for planning. Their representation quality is typically evaluated based on few-step prediction accuracy, while goal-conditioned planning aims to minimize the distance between long-term predictions and an encoded goal. In this work, we show that without proper supervision, the representation's *metric consistency* is poor, meaning that latent distances do not align well with the optimal cost of reaching the goal. We then reshape latent geometry using these costs and generalize the method through temporal ordering within recorded trajectories. Next, we show that geometry reshaping does not require extensive visual representation learning. We introduce an inexpensive test for representation sufficiency, based on the recovery of task-relevant state variables. We show empirically that once this criterion is met, visual training can stop and geometry reshaping in latent space alone can significantly improve planner performance. After two epochs of visual training in TwoRoom, the representation already satisfies our sufficiency criterion, but CEM planning success is only 11%. Freezing the encoder and continuing training in latent space raises success to 88% with proportional task-distance supervision and 90% with temporal ordering, compared with 27% using the base objective. Continued expensive visual training without reshaping plateaus at 37%.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
world models, latent-space planning, latent geometry

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ztezOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper deals with latent world models and their usage in MPC-based planning with CEM and gradient descent where the cost function is a difference in latent embeddings. The paper shows that shaping the latent space is necessary by explicitly linking it to privileged information on the ground truth distances in the problem space. They further show that when architectures are decomposed into visual encoders and adapters, training the (larger) visual encoder can be stopped early by detecting whether sufficient information (linked to privileged information on the state) is already present in the visual features.

Critical strengths

Comparing latent states directly is probably one of the most critical current weaknesses in the WM+MPC literature, so addressing the shape of the latent space is an important problem to work on.

Critical weaknesses

I would like to reassure the human co-authors of this paper that I did not use an LLM for my review and that thus their paper has not been uploaded to an LLM either. I see that as an advantage for the submitters, but it also comes with a downside: the paper needs to be understandable for a human.

NOVELTY

Shaping the latent space for planning is not a new idea and has been done in the past, even with references provided in this submission (eg. Wang et al 2026). The paper claims that these classical straightening approaches are insufficient and require task-distance supervision, but this comes with a high cost, which is unfortunately not discussed in the paper: the method actually requires access to the structure of the problem space, ie. ground truth distances between states. IMO this is an enormous drawback and it makes the method close to being handcrafted for this problem.

More generally speaking it is not clear to me what the high-level goal of this research is. As I see it, solving these planning problems (or, generally, science) can be addressed in multiple principal directions:

  • with the goal of actually solving the (application related) goal at hand, eg. navigation, manipulation, etc. and therefore optimising the performance on a downstream task which should be as complex and realistic as possible.
  • going deep into a certain methodology considered to have potential, eg. latent WMs and MPC, ignoring the fact that they are currently not competitive but trying to improve them. This can be particular interesting when this technology has more potential, has additional properties or fewer constraints than the current SoTA methods.
  • providing theoretical insights into a problem.

The current work does neither of these.

Latent WMs combined with MPC planning (like CEM) are by far not competitive at the moment, the case in point being the fact that the WM literature works on highly simplified toy problems. The TwoRooms environment is very very far from what navigation can currently do. I do not argue against research into WMs and MPC, but this submission combines this research with very application oriented constraints as the work requires actual ground truth / privileged information on the task structure. Once this information is available (and in reality it is actually never), is the proposed framework still the most obvious choice?

For comparison on realistic work on navigation, a big body of work, less recent and up to recent, trains models on realistic outdoor scenarios, and deploys on real robots. For instance the two following methods, similar to this submission, estimate distances between latent states. However, as opposed to the submission, they do NOT need ground truth distances and perform estimates from trajectories only (temporal distances).

[a] Dhruv Shah, Benjamin Eysenbach, Gregory Kahn, Nicholas Rhinehart, Sergey Levine, Rapid Exploration for Open-World Navigation with Latent Goal Models, CORL 2021. [b] Dhruv Shah, Sergey Levine, ViKiNG: Vision-Based Kilometer-Scale Navigation with Geographic Hints, RSS 2022.

If information on shortest paths (eg. the task structure) is available, it can be directly used for supervision, again, enabling navigation in real world scenarios:

[c] Kiana Ehsani et al. Imitating Shortest Paths in Simulation Enables Effective Navigation and Manipulation in the Real World, CVPR 2024.

Similarly, realistic navigation agents can now be trained 100% in simulation on the HM3D dataset and they can be deployed as is on real robots to navigate in unseen environments. This paper shows that the task structure is learned be probing planning:

[d] Janny et al., Reasoning in visual navigation of end-to-end trained agents: a dynamical systems approach, CVPR 2025.

TECHNICAL SOUNDNESS

If we accept the premise of using privileged information in the WM+MPC framework, why would the proposed method be the most natural or the best choice? Given the ground truth structure of the problem space, isn't the task of aligning it with the latent space very closely linked to estimating the value function of the task, which many methods do, most prominently critics in RL, but also other lines of work? This line of work is also mentioned in the related work section, paragraph "learning latent geometry", but it is never mentioned why the proposed direct regression should be better. It seems to me that Bellman fitting should be more principled and I don't see its downsides.

The different formulations of latent space shaping have been compared for fixed weights for their corresponding losses. It is not clear where these weights come from and whether the conclusions hold for different weights.

Equation 4: the expressions in the "s.t." section are not constraints. These expressions are merely part of the computation graph of the expression to be optimized, and the formulation is misleading is it reads like an optimization under constraints. This is not a minor error and raises further doubts on the validity of the method.

Equation (5): there is quite a big literature on learning how to rank, a discussion would have been appreciated.

Same for Temporal Ordering: there is a quite a big literature on the arrow of time and representation learning by handling the order of time steps, eg. early work by VGG:

[e] Lyndsey C. Pickup, Zheng Pan, Donglai Wei, Yichang Shih, Changshui Zhang, Andrew Zisserman, Bernhard Schölkopf and William T. Freeman, Seeing the Arrow of Time, CVPR 2014.

The three losses s,geo,order seem to be ad-hoc and lack motivation and positioning wrt the SoTa on these known problems.

Detail: L237 the loss has changed from upper case to lower case

EXPERIMENTS AND RESULTS

As already said above, the experiments are in line with the very simplistic experiments of the WM literature. While I do understand that sometimes overly simplistic environments (actually, "toy problems") can be justified to specifically test some particular line of promising models if a breakthrough is expected later, I think the community should rethink its current view on world models. These kind of models are currently held in high esteem, which entirely contrasts with the toy problems they are capable of solving. However, at some point we need to have take-away messages from papers, knowledge gained, and I am not sure what the knowledge is we gained from these experiments.

The paper also does not compare the proposed method to any SoTA method. All the comparisons are ablations of the proposed method. This fact alone would lead me to recommend rejection from any top-level conference.

PRESENTATION

While the paper is not very hard to understand, it is written in a non-conventional and very linear way. Like many papers of the very very recent literature, details and even experimental results are distributed evenly across the paper, even put into the abstract, instead of setting an appropriate level of abstraction in each section. A scientific paper should make it's case first, start with a proper motivation.

Having numerical constants directly in the method section (eg. L139) raises questions on the origin of these constants.

SUMMARY AND CONCLUSION

The paper's findings can be resumed as exploiting privileged information on the problem's structure to improve latent WMs + MPC on problems on which other methods (eg. pure RL or IL) would have a field day. Exploiting privileged information can be extremely helpful, but it is application driven and normally employed when we want to optimize performance on a downstream task. It is unsure what it tells us for latent WMs, which are generally considered as the way to scale up learning on large amount of unlabelled data, in the wild.

The work does not compare to the SoTA, even on the single simple task it evaluates on.

Questions for authors

See "weaknesses" section.

8 comments
AuthorsOpenReview ↗

Response to Reviewer ztez (1/6)

We thank the reviewer for the careful and detailed review of the paper. We address all raised points in the comments below. To avoid repetition we kindly ask the reviewer to find the details of added experiments in the official comment at the top of the page.

I would like to reassure the human co-authors of this paper that I did not use an LLM for my review and that thus their paper has not been uploaded to an LLM either. I see that as an advantage for the submitters, but it also comes with a downside: the paper needs to be understandable for a human.

We appreciate the transparency, and likewise assure the reviewer that our paper and responses are human-generated.

NOVELTY

Shaping the latent space for planning is not a new idea and has been done in the past, even with references provided in this submission (eg. Wang et al 2026). The paper claims that these classical straightening approaches are insufficient and require task-distance supervision, but this comes with a high cost, which is unfortunately not discussed in the paper: the method actually requires access to the structure of the problem space, ie. ground truth distances between states. IMO this is an enormous drawback and it makes the method close to being handcrafted for this problem.

We agree that adding ground-truth distances to training would require having significantly more privileged knowledge than the temporal straightening technique of Wang et al. 2026. This was precisely our motivation to also propose the supervision technique with Lorder\mathcal{L} _ {\mathrm{order}}, which requires no such information. However, we acknowledge that we may not have made this point clear enough, and that by training the adapter method on a mix of suboptimal and optimal (A*-generated) plans may have colluded effects of data vs. method, which reviewer tyNZ also pointed out. We are thus suggesting the following changes:

  • In lines 224 of the paper, we suggest adding the following clarification (in bold):

Temporal ordering uses the order of states within recorded trajectories as an alternative source of geometric supervision. This assumes that later states tend to be closer to the trajectory endpoint. Note that the plans need not be optimal as for Lgeo\mathcal{L} _ {\mathrm{geo}} -- we found empirically that it suffices that a sufficient number of recorded trajectories are goal-directed.

  • As described in our response to Reviewer tyNZ (W1, Table 2), we have added a new experiment that disentangles the effect of the optimality of the dataset from the latent reshaping. Interestingly, we found that temporal straightening by Wang et al. 2026 benefits strongly from optimal datasets (from 43.3% to 74.0%), although our method still outperforms it there (81.9%). However, the main advantage of our method with Lorder\mathcal{L} _ {\mathrm{order}} is that it has much weaker requirements on the optimality of the provided plans, so it also works on the suboptimal dataset (88.7%), where temporal straightening does significantly worse (43.3%).

  • We hope that the addition of the new benchmark, as described in our response to Reviewer HUKx (Q1; +21.4 points on Cube), also underlines that the method is not merely handcrafted for this particular use case. We do acknowledge that there are limitations to the approach, and suggest to add the following in the Discussion section of the paper:

    The proposed method shows significant improvements on tasks with limited complexity. To generalize across more complex robotics tasks, such as pick-and-place tasks or underactuated pushing tasks, we expect that additional measures may be needed for success, such as combining a subtask-dependent reshaping with higher-level, contact-dependent, plans.

  • The second technical contribution of the paper, the sufficiency test, does require ground-truth state information. We acknowledge that this privileged information may not always be available (but quite often it is, see remark below), so this part of the contribution does not always apply. We propose the following addition in the limitations section:

The sufficiency criterion currently requires privileged information in the form of a task-relevant state representation. Investigating weaker criteria for settings where such information is difficult to obtain is left for future work. A promising direction, inspired by the geometry supervision, could be to measure the ability to predict temporal order.

While we hope that these points address the reviewer's concerns, we do want to point out that we believe that privileged information in the form of optimal distances, optimal plans, etc., is not uncommonly avialable when distilling world models from simulators. For robotics problems, this is a valuable usecase as it still allows for creating fast and reactive policies at runtime, with privileged information only used at training time.

AuthorsOpenReview ↗

Response to Reviewer ztez (2/6)

More generally speaking it is not clear to me what the high-level goal of this research is. As I see it, solving these planning problems (or, generally, science) can be addressed in multiple principal directions:

  • with the goal of actually solving the (application related) goal at hand, eg. navigation, manipulation, etc. and therefore optimising the performance on a downstream task which should be as complex and realistic as possible.

  • going deep into a certain methodology considered to have potential, eg. latent WMs and MPC, ignoring the fact that they are currently not competitive but trying to improve them. This can be particular interesting when this technology has more potential, has additional properties or fewer constraints than the current SoTA methods.

  • providing theoretical insights into a problem.

The current work does neither of these.

We agree with this high-level classification of research/scientific contributions. Our goal is to be contributing along point 2) (improvements of latent WMs) and not 1) (improving real-world navigation solutions).

We believe that this combination (WMs and optimization in latent space) has great potential for many applications, for the following reasons.

  • Using MPC in latent space inherits one of the main advantages of MPC over policy learning: New behaviors can be generated at deployment time simply be changing the objective of the MPC problem, there is no need to train a new policy for every new task.
  • While it is currently only used for planning in our paper, optimization in latent space has been shown to have potential for other tasks, for example latent safety filters
  • Even if policy learning is the method of choice, there is existing evidence that model-based MPC, for example in the form of a learned dynamics model in latent space, is more sample-efficient than model-free reinforcement learning.

Therefore, we argue that investigation of latent world models like JEPA is of value even if currently domain and task-specific solutions outperform it. In particular, our paper

  1. shows and quantifies an important problem of latent space geometry when used for planning
  2. provides a methodology to fix these latent spaces that is fairly general, computationally cheap, and does not require extensive engineering effort (unlike, e.g., training value functions that can adapt or compensate for the unfavorable geometry of the latent space).

Latent WMs combined with MPC planning (like CEM) are by far not competitive at the moment, the case in point being the fact that the WM literature works on highly simplified toy problems. The TwoRooms environment is very very far from what navigation can currently do. I do not argue against research into WMs and MPC, but this submission combines this research with very application oriented constraints as the work requires actual ground truth / privileged information on the task structure. Once this information is available (and in reality it is actually never), is the proposed framework still the most obvious choice?

  • As argued in the previous reply, the regularization with Lorder\mathcal{L} _ {\mathrm{order}} does not require strong priviledged information, and its applicability is not limited to simple planar navigation, as the new results in the Cube benchmark show.
  • That the proposed framework is the most obvious choice is not our claim. However, we do want to repeat the point from above that learning a latent space with metrically consistent structure is of broader interest and relevance, even if a dedicated task-based methods may outperform this particular method in terms of planning preformance.
AuthorsOpenReview ↗

Response to Reviewer ztez (3/6)

For comparison on realistic work on navigation, a big body of work, less recent and up to recent, trains models on realistic outdoor scenarios, and deploys on real robots. For instance the two following methods, similar to this submission, estimate distances between latent states. However, as opposed to the submission, they do NOT need ground truth distances and perform estimates from trajectories only (temporal distances).

[a] Dhruv Shah, Benjamin Eysenbach, Gregory Kahn, Nicholas Rhinehart, Sergey Levine, Rapid Exploration for Open-World Navigation with Latent Goal Models, CORL 2021. [b] Dhruv Shah, Sergey Levine, ViKiNG: Vision-Based Kilometer-Scale Navigation with Geographic Hints, RSS 2022.

If information on shortest paths (eg. the task structure) is available, it can be directly used for supervision, again, enabling navigation in real world scenarios:

[c] Kiana Ehsani et al. Imitating Shortest Paths in Simulation Enables Effective Navigation and Manipulation in the Real World, CVPR 2024.

Similarly, realistic navigation agents can now be trained 100% in simulation on the HM3D dataset and they can be deployed as is on real robots to navigate in unseen environments. This paper shows that the task structure is learned be probing planning:

[d] Janny et al., Reasoning in visual navigation of end-to-end trained agents: a dynamical systems approach, CVPR 2025.

  • We agree that these papers show impressive real-world performance of navigation solutions, but our point is not to focus on navigation per se. The proposed methodology from the papers does not transfer to, for example, robotic manipulation experiments, while WM+MPC promises this kind of generalization. We acknowledge that WM+MPC does currently have limitations, such as the latent geometry, which is exactly what we are aiming to show and improve on in this submission. We will make this clearer in our paper.

TECHNICAL SOUNDNESS

If we accept the premise of using privileged information in the WM+MPC framework, why would the proposed method be the most natural or the best choice? Given the ground truth structure of the problem space, isn't the task of aligning it with the latent space very closely linked to estimating the value function of the task, which many methods do, most prominently critics in RL, but also other lines of work? This line of work is also mentioned in the related work section, paragraph "learning latent geometry", but it is never mentioned why the proposed direct regression should be better. It seems to me that Bellman fitting should be more principled and I don't see its downsides.

  • Yes, learning a critic, at least for goal-conditioned reinforcement learning, is directly related to learning the cost-to-go.

However, in our setting, because we assume that the ground-truth distance is known, we regress latent distances directly onto it, while Bellman fitting estimates the cost-to-go by bootstrapping from the current estimate. Bootstrapping propagates errors from the current estimate and adds hyperparameters, and Destrade et al. (2026), who learn such a critic, report training instabilities and inaccurate value functions. We argue that direct regression is the natural choice when privileged information is available.

However, the advantage of the proposed method is that it comes at a lower computational cost and engineering effort. Learning a reliable critic comes with its own challenges, as Destrade et al. (2026), who propose to do exactly that, note. By simply using the temporal ordering, such training difficulties and costs can be avoided.

  • Furthermore, learning a good latent geometry is interesting because of other optimization-based tasks that this may unlock in latent space, although the validation of this thesis remains for future work.
Reviewer_tyNZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how to make distances between world-model representations more useful for goal-conditioned planning. Using LeWorldModel, the authors freeze the visual encoder and jointly train a small residual adapter and the dynamics predictor. They add either a loss that fits latent distances to A* shortest-path lengths or a ranking loss that encourages later states along a trajectory to lie closer to its endpoint. The temporal-ordering experiment uses both recorded trajectory segments and A* paths. The authors also investigate position-decoding accuracy as a criterion for deciding when visual training can stop. In TwoRoom, adaptation from an encoder trained for two epochs raises CEM planning success from 11%11\% to approximately 88−90%88-90\%, compared with 27%27\% after adaptation using the base objective. The evaluation includes several visual-training checkpoints, three adaptation seeds, and a comparison using Adam for planning.

Critical strengths
  1. The comparison using the same frozen encoder is the most convincing part of the paper. In Section 4.14.1, the authors keep the visual features fixed and compare several objectives for training the adapter and predictor. CEM planning success reaches 88.1% with shortest-path supervision and 89.8%89.8\% with temporal ordering, compared with 27.4%27.4\% using the base objective. Figures 1−41-4 also show improved agreement between latent distances and navigation costs. Together, these results demonstrate that the early visual features can support much better planning when the subsequent training objective is changed.

  2. The evaluation goes beyond a single favorable comparison. Figure 66 examines four visual-training checkpoints, while Section 4.14.1 repeats adaptation with three seeds and reports improvements with both CEM and Adam planning. The comparison with temporal straightening is also useful because it tests another established way of modifying representations for planning. These checks strengthen the evidence for the combined adapter-and-predictor training procedure, even though its evaluation remains confined to TwoRoom.

Critical weaknesses
  1. The temporal-ordering result leaves the contribution of the objective difficult to separate from the information supplied by A* paths. Section 3.3.33.3.3 motivates the loss by assuming that later states along a trajectory tend to be closer to its endpoint. Shortest paths naturally satisfy this assumption, whereas recorded trajectories may contain detours or reversals. However, Section 4.14.1combines recorded windows with A*-generated paths, so the reported 89.8%89.8\% success does not establish how effective the objective is with recorded trajectories alone. Recorded-only and A*-only results, together with a straightening baseline trained on the same paths, would clarify whether the gain comes primarily from the ranking objective or the additional navigation information. Section 55 acknowledges the use of map information, but the current comparison leaves this central distinction unresolved.

  2. The evidence for the proposed stopping criterion is preliminary. Section 3.43.4 defines the criterion using median position-decoding error below one pixel, and Figures 5−65-6 show that crossing this threshold coincides with a substantial improvement in subsequent adaptation. However, the visual checkpoints come from one training run; the three adaptation seeds do not test whether the threshold reliably selects a useful checkpoint in independently trained encoders. This matters because the paper proposes the probe as a practical decision rule for stopping visual training. Moreover, Figure 66 compares optimizer updates rather than total computation, and Section 55 acknowledges that wall-clock costs remain unmeasured. Thus, both the reliability of the decision rule and the size of the computational saving remain uncertain.

Questions for authors
  1. (W1) What happens with recorded-only, A*-only, and combined temporal-ordering supervision? Does straightening improve when supplied with the same A* paths? These comparisons would distinguish the benefit of the objective from the additional path information.

  2. What is the end-to-end computational cost, including probe fitting, feature caching, and adaptation, relative to continued visual training?

5 comments
AuthorsOpenReview ↗

Response to Reviewer tyNZ (1/2)

We thank the reviewer for the positive review and constructive suggestions. We address all raised points in the comments below. To avoid repetition we kindly ask the reviewer to find the details of added experiments in the official comment at the top of the page.

The temporal-ordering result leaves the contribution of the objective difficult to separate from the information supplied by A* paths. Section 3.3.33.3.3 motivates the loss by assuming that later states along a trajectory tend to be closer to its endpoint. Shortest paths naturally satisfy this assumption, whereas recorded trajectories may contain detours or reversals. However, Section 4.14.1combines recorded windows with A*-generated paths, so the reported 89.889.8 success does not establish how effective the objective is with recorded trajectories alone. Recorded-only and A*-only results, together with a straightening baseline trained on the same paths, would clarify whether the gain comes primarily from the ranking objective or the additional navigation information. Section 55 acknowledges the use of map information, but the current comparison leaves this central distinction unresolved.

The evidence for the proposed stopping criterion is preliminary. Section 3.43.4 defines the criterion using median position-decoding error below one pixel, and Figures 5−65−6 show that crossing this threshold coincides with a substantial improvement in subsequent adaptation. However, the visual checkpoints come from one training run; the three adaptation seeds do not test whether the threshold reliably selects a useful checkpoint in independently trained encoders. This matters because the paper proposes the probe as a practical decision rule for stopping visual training. Moreover, Figure 66 compares optimizer updates rather than total computation, and Section 55 acknowledges that wall-clock costs remain unmeasured. Thus, both the reliability of the decision rule and the size of the computational saving remain uncertain.

We thank the reviewer for this point. We tested the criterion on two new encoders trained from scratch with different initialization seeds (Table 5b). On both, the probe first falls below 1 px at 2,500 updates, precisely when success after adaptation jumps (by 25 to 38 points from 2,000 updates, for both Lgeo\mathcal{L} _ {\mathrm{geo}} and temporal ordering); however, Lgeo\mathcal{L} _ {\mathrm{geo}} then reaches only 46.3% and 38.3%, against 69.0% and 78.3% at 3,000 updates. Crossing 1 px therefore marks the onset of the improvement rather than a ready checkpoint. In the revised paper, we will report all checkpoints and correct the caption of Figure 6, whose checkpoints come from two training runs.

AuthorsOpenReview ↗

Response to Reviewer tyNZ (2/2)

(W1) What happens with recorded-only, A*-only, and combined temporal-ordering supervision? Does straightening improve when supplied with the same A* paths? These comparisons would distinguish the benefit of the objective from the additional path information.

We thank the reviewer for pointing out this issue. To separate the gain coming from temporal-ordering supervision from the additional information added by using optimal trajectories, we have added new experiments where we trained our adapation layer separately on either A* trajectories or on the recorded trajectories provided by LeWorldModel (suboptimal goal-directed trajectories generated by a noisy heuristic policy)

We report the results in Table 2, with the same frozen encoder and three adapter seeds.

Temporal ordering trained exclusively on recorded trajectories achieves 88.7% success, compared with 87.3% when combining them with A* trajectories, a difference of +1.3 points [−3.9, 6.6]. This shows that the additional information provided by optimal trajectories is not necessary to obtain the improvement, and that temporal ordering can work directly on suboptimal goal-directed trajectories without requiring access to the map.

Finally, we also tested the temporal straightening baseline trained on A* paths. Interestingly, its performance increases from 43.3% on recorded trajectories to 74.0% on A* paths, showing that straightening benefits significantly from the additional path information. Temporal ordering achieves 81.9% on these same A* trajectories, outperforming straightening by 7.9 points [1.9, 13.8]. Importantly, the performance of temporal straightening drops drastically when training on recorded trajectories only. Temporal ordering, on the other hand, even when trained only on recorded trajectories, outperforms straightening trained on A* paths by 14.7 points [8.7, 20.8].

Overall, these experiments show that optimal A* trajectories help temporal straightening but are not needed for temporal ordering: trained on recorded (suboptimal) trajectories alone, temporal ordering performs as well as with A* paths and outperforms straightening even when straightening is trained on A* paths. We will include these findings in the revised paper.

What is the end-to-end computational cost, including probe fitting, feature caching, and adaptation, relative to continued visual training?

We thank the reviewer for raising this question. We measured the end-to-end wall-clock time of our method on a single RTX 5090, including visual training, feature caching, and both adaptation stages. We also measured the computational cost of the position-decoding probe. Detailed results are reported in Tables 6a–b.

The adaptation itself is inexpensive. In both stages, the visual encoder is frozen: the first stage trains only the predictor with the base objective, while the second jointly trains the adapter and predictor with the additional geometric regularization. Because the image features are computed once and cached, adaptation updates never run the visual encoder. The whole adaptation therefore takes approximately 6.5 minutes, and temporal ordering on recorded trajectories requires no A* paths.

Overall, visual training to epoch 2 followed by temporal ordering reaches 88–90% success in 53 minutes, compared with 33–35% after 3 hours and 49 minutes of continued visual training alone. Our approach is therefore approximately 4.3 times faster end to end, with substantially better planning performance. The 53-minute pipeline uses the predefined epoch-2 checkpoint, while the computational cost of probe measurements is reported separately in Table 6a.

Dear Reviewer,

Thank you for your efforts in reviewing this submission. As the discussion period is approaching its end, I would like to kindly remind you to read the authors' rebuttal and engage in the discussion. Your input is important for reaching a well-informed final assessment.

Thank you for your time and cooperation.

Reviewer_HUKxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper notes latent world models are trained to predict accurately, but planners use latent distances to rank outcomes. These two objectives don't align usually. If latent distances don't match actual task distances (how far states are from each other along feasible paths), the planner will consistently choose the wrong action. This is not a problem with the planner per se, but it is a problem with the geometry of the latent space not aligning with the geometry of the task space (or cost space perhaps).

The authors call this metric consistency: a latent space is metrically consistent if Euclidean distances in latent space preserve the same ranking as optimal task distances. They propose to fix it not by retraining the visual encoder, but by training a lightweight residual adapter on top of a frozen encoder. After two epochs of visual training, continued visual training without reshaping plateaus at 37% planning success, while freezing the encoder and reshaping latent geometry alone reaches 88–90% on TwoRoom long-range navigation. Substantial results.

Critical strengths
  1. Studies a widely known phenomenon but not yet rigorously looked at. This was a problem that is largely known to practitioners. But, it has not been studies extensively. The paper offers one of the first examples of a clear look at the problem and tries to identify what's missing.

  2. Strong empirical gains. The gains reported in the paper seem to be significant. The gap between continued visual training (37% plateau) and cheap latent-space reshaping (88–90%) is striking. I would not have expected such a relatively large gain. (Although I am not sure whether we understand this metric correctly either. More on that below.)

Critical weaknesses
  1. Biggest problem is the severely limited single-environment study. It is unclear how the empirical results generalize.

  2. If I understand correctly, the supervised variant depends on map information. It may be limiting factor in many applications. The authors address this with L_order. However, the conditions under which L_order succeeds does not seem to be well characterized.

Questions for authors

Q1: What are ways to ensure that there is indeed generalization? Could there be experiments on additional environments, scenarios?What do we expect?

Q2: The success rate is quite high in the benchmark. However, I had a feeling that the benchmark has almost been constructed to provide a generous success rate to the proposed method. We have a 2d maze, 150 action budget (which may be generous), and adding 300 tasks, min-max bands, even the 3 pixel radius, make me think how sensitive is the success rate to these?

4 comments
AuthorsOpenReview ↗

Response to Reviewer HUKx (1/2)

We thank the reviewer for the positive review and constructive suggestions. We address all raised points in the comments below. To avoid repetition we kindly ask the reviewer to find the details of added experiments in the official comment at the top of the page.

Biggest problem is the severely limited single-environment study. It is unclear how the empirical results generalize.

If I understand correctly, the supervised variant depends on map information. It may be limiting factor in many applications. The authors address this with L_order. However, the conditions under which L_order succeeds does not seem to be well characterized.

Q1: What are ways to ensure that there is indeed generalization? Could there be experiments on additional environments, scenarios?What do we expect?

The question of generalization of our method is very important. The geometric regularization Lgeo\mathcal{L} _ {\mathrm{geo}} was designed as a case study for the TwoRoom environment, while its more general extension, temporal ordering Lorder\mathcal{L} _ {\mathrm{order}}, was designed to be task-agnostic. We extented our proposed temporal ordering using only recorded trajectories (no A* paths) to two additional environments: Cube and PushT. The first involves a robotic arm moving a cube toward a target position, while the second involves a T-shaped object and a pusher whose goal is to move the object to a target position and orientation.

On Cube, for long-range tasks, we observe a clear improvement over the baseline: +21.4 percentage points [16.3, 26.7] at the 4 cm success threshold, and +7.1 points [2.8, 11.3] at the 2 cm threshold (95% confidence intervals; Table 1). On PushT, we have not observed an improvement so far; we discuss a likely reason below.

Conditions under which Lorder\mathcal{L} _ {\mathrm{order}} succeeds. Temporal ordering arranges the embeddings within each window so that each later state is closer to the endpoint than each earlier one. This ordering is consistent in TwoRoom and Cube, where the agent or the arm moves toward its target without coming back. In PushT, the pusher often has to reposition itself around the T before pushing it further, and may return to a state nearly identical to an earlier one in the same window. The loss then requires two nearly identical observations to lie at different distances from the endpoint, which conflicts with mapping similar observations to similar embeddings. This remains a hypothesis; we will add this discussion to the paper. The other condition we identified, goal-directed but possibly suboptimal trajectories, is discussed in our reply to Reviewer tyNZ (Table 2).

AuthorsOpenReview ↗

Response to Reviewer HUKx (2/2)

Q2: The success rate is quite high in the benchmark. However, I had a feeling that the benchmark has almost been constructed to provide a generous success rate to the proposed method. We have a 2d maze, 150 action budget (which may be generous), and adding 300 tasks, min-max bands, even the 3 pixel radius, make me think how sensitive is the success rate to these?

We thank the reviewer for raising this question. We agree that choosing appropriate evaluation criteria is fundamental. Below, we first justify the choices that we made, and then discuss additional ablations that we have added.

In the original TwoRoom benchmark used by LeWM (Maes et al., 2026), the goal is sampled 25 steps ahead, with a budget of 50 actions and a success radius of 16 px. We made the task more challenging by reducing the radius to 3 px and increasing the start-to-goal separation to 75 steps, increasing the action budget proportionally to 150.

To validate this choice, we have now conducted a systematic sensitivity analysis over the action budget, success radius, and start-to-goal distance.

In Table 3a, we show that the budget of 150 actions is indeed generous, representing 2.4 to 11 times the direct path length depending on the task. But it is not necessary for the improvement. For tasks wih a longer distance to the goal than 100 px, the baseline remains below 7% success regardless of the budget, while temporal ordering, at 100-150px distance, reaches at least 72.9% for 75 actions or more. Even with a budget of less than twice the direct path length, the gap is already clear: 75.8% versus 22.2% at 50 actions for tasks between 50 and 100 px.

In Table 3b, we show that the improvements of our method also persist as we change the success radius. The performance gap persists from a 1 px radius (57.0% versus 19.1%) to a 16 px radius (99.8% versus 29.9%). Increasing the radius from 3 to 16 px improves the baseline by only 2.6 points, suggesting that its failures are not simply near misses.

Regarding the selection of the 300 evaluation tasks, we generated a new panel of 300 tasks after receiving the reviews. The results remain consistent: 27.3% for the baseline and 87.3% for temporal ordering, compared with 27.4% and 89.8% on the original task set (Table 2). Reweighting the tasks to the natural distance distribution also preserves the improvement (+70.4 points).

Finally, the min–max bands in our figures represented the minimum and maximum across three adapter seeds. We now additionally report 95% confidence intervals using a two-level bootstrap over tasks and adapter seeds.

Overall, these additional experiments show that the improvement is robust to the benchmark design choices highlighted by the reviewer.

Dear Reviewer,

Thank you for your efforts in reviewing this submission. As the discussion period is approaching its end, I would like to kindly remind you to read the authors' rebuttal and engage in the discussion. Your input is important for reaching a well-informed final assessment.

Thank you for your time and cooperation.

AuthorsOpenReview ↗

Revised paper

Thank you again for your review. The revised paper now includes the experiments from our reply: Cube (Appendix C.2), sensitivity to the action budget and the success radius (Appendix C.4), a second set of 300 tasks (Appendix C.1), and 95% confidence intervals (Appendix C). These analyses use temporal ordering trained on recorded trajectories only, as in the main text; the conclusions of our reply are unchanged. Section 5 now discusses the conditions under which temporal ordering succeeds. We hope this addresses your concerns.

Comments on the paper

4 comments
AuthorsOpenReview ↗

General Response to All Reviewers and Summary of Added Experiments (1/3)

We thank all three reviewers for their careful reading of our paper and for their constructive comments, and the Area Chair for handling the submission.

We understand the reviewers' main concerns regarding the generalization of our method, and the request for more careful ablations regarding the role of privileged information and other elements of the proposed solutions are justified.

To address these concerns, we conducted extensive additional experiments. To sum up the most important changes:

  • We extended our evaluation to the Cube, where the method also achieves signifcant gains over the baseline. This shows the generalization capability of the method, and we will add these new results in an updated version of the paper. We have not observed an improvement on PushT so far, which we list in the limitations section of the revised paper along with some potential explanations and avenues for future work. This answers to questions raised by reviewers HUKx and ztez and the details are listed in Table 1 below.

  • We improved the empirical evaluation by separating the effect of temporal ordering from the additional information provided by A* trajectories. This was explicitly asked for by reviewer tyNZ and clarifies the role of privileged information criticized by reviewer ztez. These experiments have provided some interesting new insights into why temporal ordering works better than temporal straightening, which we will include in the paper. Details are provided in Table 2 below.

  • We investigated the sensitivity to the action budget, success radius, and evaluation tasks (reviewer HUKx), and regularization weights (reviewer ztez). Details are provided in Tables 2, 3a, 3b, and 4. These new ablations will be listed in full in the Appendix of the revised paper and summarized in the main text. They justify the choices made and will provide more guidance for people building on the method.

  • Finally, as suggested by reviewer tyNZ, we also tested our stopping criterion on two independently trained encoders (Tables 5a and 5b), and report the end-to-end computational cost of our method compared to continued visual training in Tables 6a and 6b. As expected, our method is significantly cheaper than continued visual training, while providing higher planning success rates. We will add the encoder ablation to the Appendix and report the numerical timing results in the main text of the paper.

The paper's task set is the set of 300 TwoRoom tasks used in the submission. To check that our results do not depend on this particular set, we drew a new task set of 300 tasks after the reviews, with the same protocol (goal 75 steps ahead, 150 actions, 3 px radius) and drawn by 50 px bands of shortest-path distance, which also enables the per-distance analysis of Table 3a. Both sets give the same picture (Table 2); new results are reported on the new task set unless stated otherwise. Numbers are means over three adapter training runs unless stated otherwise, and brackets give 95% confidence intervals from a two-level bootstrap over tasks and runs.

We greatly appreciate the reviewers' insights and believe these additional experiments strengthen the paper significantly. Below, we list the requested experiments by reviewer and report them in tables whose captions give the experimental details. We then address each reviewer's comments individually, referring to these tables to avoid repetition.

Overview of requested experiments

ReviewerRequestTable
1: HUKxEvaluation on other environments1
1: HUKxConditions under which Lorder\mathcal{L}_{\mathrm{order}} works2
1: HUKxSensitivity to the action budget3a
1: HUKxSensitivity to the success radius3b
1: HUKxSensitivity to the choice of the 300 tasksNew task set in Tables 2–4
1: HUKxMin–max bands95% confidence intervals (Table 1 and replies)
2: tyNZLorder\mathcal{L}_{\mathrm{order}} on recorded only, A* only and both; straightening on the same A* paths2
2: tyNZStopping criterion on independently trained encoders5a, 5b
2: tyNZEnd-to-end compute cost6a, 6b
3: ztezSensitivity to the loss weights4
3: ztezMethod without privileged information2
3: ztezComparison to Bellman fittingOmited with justficiation, see our reply
AuthorsOpenReview ↗

General Response to All Reviewers and Summary of Added Experiments (2/3)

Table 1. Temporal ordering also improves planning on Cube. Base objective versus temporal ordering trained on recorded trajectories only (no A* paths), on the same frozen visual encoder. TwoRoom: our encoder, new task set, goal 75 actions ahead. Cube: published LeWM encoder, goal 150 actions ahead. Mean of three adapter runs; brackets: 95% confidence interval on the difference.

EnvironmentSuccess criterionBase (%)Lorder\mathcal{L} _ {\mathrm{order}} (%)Difference [95% CI]
TwoRoomwithin 3 px27.388.7+61.3 [+55.2, +67.3]
Cubewithin 4 cm14.836.2+21.4 [+16.3, +26.7]
Cubewithin 2 cm10.817.9+7.1 [+2.8, +11.3]

Table 2. Temporal ordering does not need the A* paths; straightening does. Each objective trains the adapter and predictor on the same frozen encoder (epoch 2), using recorded trajectories (noisy goal-directed trajectories of the LeWM dataset), A* shortest paths computed on the map, or both. Mean of three adapter runs.

ObjectiveTraining dataPaper's task set (%)New task set (%)
Base—27.427.3
Lorder\mathcal{L} _ {\mathrm{order}}recorded + A* (submission)89.887.3
Lorder\mathcal{L} _ {\mathrm{order}}recorded only91.688.7
Lorder\mathcal{L} _ {\mathrm{order}}A* only85.081.9
Ls\mathcal{L} _ {\mathrm{s}}recorded (submission)42.143.3
Ls\mathcal{L} _ {\mathrm{s}}A* only75.674.0

Table 3a. The gain does not depend on the generous action budget. Success (%) of the base objective / temporal ordering, by shortest-path distance between start and goal and by action budget. Success radius 3 px, new task set, mean of three adapter runs. Direct trip: median path length of the band divided by the agent's median speed (3.3 px per action).

Shortest pathTasksDirect trip25 actions50 actions75 actions100 actions150 actions
0–50 px66≈ 14 actions35.9 / 47.569.2 / 80.374.2 / 89.977.3 / 91.980.8 / 94.9
50–100 px69≈ 28 actions3.9 / 15.922.2 / 75.830.9 / 87.431.9 / 89.933.3 / 93.2
100–150 px69≈ 36 actions0.0 / 1.41.4 / 52.22.9 / 72.93.9 / 77.86.8 / 84.5
150–200 px85≈ 48 actions0.0 / 0.00.0 / 32.20.4 / 60.80.4 / 69.41.2 / 80.4
200+ px11≈ 62 actions0.0 / 0.00.0 / 6.10.0 / 36.40.0 / 54.50.0 / 75.8
All3008.8 / 14.420.7 / 56.424.2 / 75.225.3 / 80.427.3 / 87.3

Table 3b. The gain holds for every success radius. Success (%) with 150 actions, all distances, new task set, mean of three adapter runs.

Success radius1 px2 px3 px5 px8 px16 px
Base19.125.327.329.229.629.9
Lgeo\mathcal{L} _ {\mathrm{geo}}53.077.384.394.696.897.3
Lorder\mathcal{L} _ {\mathrm{order}} (recorded + A*, submission)57.078.887.396.399.499.8
Lorder\mathcal{L} _ {\mathrm{order}} (recorded only)58.981.188.797.798.898.8

Table 4. Weight sweep: λ=3\lambda=3 is at or near the best value for Lgeo\mathcal{L} _ {\mathrm{geo}} and Lorder\mathcal{L} _ {\mathrm{order}}. Success (%) on the new task set for each regularization weight in {0.3,1,3,10}\lbrace 0.3, 1, 3, 10 \rbrace, all other settings unchanged; mean of three adapter runs.

Weight0.31310
Ls\mathcal{L} _ {\mathrm{s}}40.143.349.346.4
Lgeo\mathcal{L} _ {\mathrm{geo}}75.866.384.385.7
Lorder\mathcal{L} _ {\mathrm{order}}64.487.187.379.9
AuthorsOpenReview ↗

General Response to All Reviewers and Summary of Added Experiments (3/3)

Table 5a. Submission checkpoints: crossing 1 px is not sufficient on its own. The eight visual checkpoints we adapted. Median position-decoding error of the probe, and planning success after adaptation with Lgeo\mathcal{L} _ {\mathrm{geo}} (paper's task set, mean of three adapter runs). With temporal ordering, success is 44.3% from epoch 1 and 89.8% from epoch 2.

Visual checkpoint (updates)Position-decoding error (px)Lgeo\mathcal{L} _ {\mathrm{geo}} success (%)
2,0001.4612.1
2,5001.0933.1
3,0000.54578.3
5,138 (epoch 1)0.63646.6
10,276 (epoch 2)0.66188.1
15,414 (epoch 3)0.5682.3
20,552 (epoch 4)0.4974.6
25,690 (epoch 5)0.4671.4

Table 5b. Two new visual encoders trained from scratch, with different initialization seeds and the same data order; one adaptation run per checkpoint. Success (%): paper's task set / new task set.

Init. seedUpdatesPosition-decoding error (px)BaseLgeo\mathcal{L} _ {\mathrm{geo}}Lorder\mathcal{L} _ {\mathrm{order}} (recorded only)
1012,0001.4029.7 / 27.012.3 / 8.727.3 / 28.3
1012,5000.8823.0 / 18.341.0 / 46.352.7 / 53.3
1013,0000.5226.3 / 22.370.7 / 69.060.3 / 63.0
10110,276 (epoch 2)0.6828.7 / 30.764.7 / 64.091.0 / 88.3
1022,0001.2217.7 / 18.319.7 / 12.739.3 / 39.0
1022,5000.6427.3 / 24.741.7 / 38.375.7 / 75.7
1023,0000.6427.7 / 28.781.3 / 78.386.0 / 80.3
10210,276 (epoch 2)0.3731.3 / 33.073.0 / 71.391.0 / 90.3

Table 6a. Cost of the position-decoding probe (one RTX 5090).

MeasurementTime
One measurement, images loaded once and kept in memory2.1 s, after ≈70 s of initial loading

Table 6b. Our pipeline is about 4.3 times cheaper than continued visual training. Wall-clock time on one RTX 5090, including start-up, visual training, feature caching and both adaptation stages; final evaluation excluded. Success range over the two new encoders, new task set.

PipelineTimeSuccess (%)
Visual training to the first checkpoint below 1 px (2,500 updates), then Lorder\mathcal{L} _ {\mathrm{order}}, probe measurements included20 min53–76
Visual training to epoch 2 (10,276 updates), then Lorder\mathcal{L} _ {\mathrm{order}}53 min88–90
Visual training to epoch 10 (51,380 updates), no adaptation3 h 49 min33–35
AuthorsOpenReview ↗

Revised PDF uploaded

We have uploaded a revised version of the paper. All changes are marked in blue. Main changes:

  • Other environments (HUKx, ztez): temporal ordering also improves planning on Cube, from 14.8% to 36.2% success (Section 4.1, Appendix C.2). PushT is discussed in the limitations.
  • Privileged information (tyNZ, ztez): trained on recorded trajectories only, without A* paths, temporal ordering reaches 89.6%, while straightening stays below even with A* paths (Section 4.1, Appendix C.3).
  • Evaluation protocol (HUKx, ztez): results for other action budgets, success radii, loss weights and a second set of 300 tasks, with 95% confidence intervals (Appendices C.1, C.4 and C.5).
  • Stopping criterion (tyNZ): all eight adapted checkpoints and two visual encoders trained from scratch (Appendix C.6). The probe is now described as an early indicator that adaptation may be worthwhile.
  • Computational cost (tyNZ): end-to-end wall-clock times (Section 4.2, Appendix C.7).
  • Clarity (ztez): hyperparameters moved out of the method section, added references (Kendall, Joachims, RankNet), and an expanded Discussion and Limitations section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.