acceptodds
Withdrawn from ICLR 2027

Disentangling the Geometries of Prediction and Planning in Latent World Models through Trajectory-Graph Control Calibration

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Joint-Embedding Predictive Architectures (JEPAs) provide an efficient foundation for latent world models by predicting future representations rather than reconstructing pixels. However, Model Predictive Control (MPC) planners built on these models typically select action sequences by minimizing the terminal latent distance to the goal, which implicitly assumes that latent similarity measures the remaining cost to the goal. In contrast, goal-reaching cost can be directional, violating the symmetry imposed by latent-distance planning and potentially causing the planner to prematurely discard dynamically preferable action sequences. To address this limitation, we propose Trajectory-Graph Control Calibration (TGCC), a lightweight post-hoc method that disentangles the geometries of prediction and planning over the same frozen latent representations. Concretely, TGCC constructs a directed graph over offline trajectories and distills its temporal relations into a control geometry network. During planning, the network's graph-derived control score is combined with terminal latent similarity to rank candidate action sequences, thereby defining a planning geometry beyond the prediction geometry. Without retraining the world model, TGCC achieves superior planning performance across multiple datasets and latent world-model backbones, with higher success rates and fewer steps to success.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Latent world model, Disentanglement, Graph

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_azHiOpenReview ↗
Rating
1(−1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

In LeWM and PLDM, the planning objective is the distance between the predicted final state and the goal. This objective is symmetric and blind to obstacles, which leads to poor planning performance. The authors propose TGCC, a post-hoc add-on that keeps the world model frozen. It builds a directed graph from offline trajectories, uses BFS to label which state pairs are reachable, and distills these labels into a small network that serves as a control cost. The authors show that with this cost, the world model's planning performance improves.

Critical strengths
  1. The paper proposes a lightweight, general fix that could be applied anywhere.
  2. Experiments on the benchmarks show that it matches or beats all baselines across four tasks.
  3. The paper provides clean supporting analysis in Two-Room, and the ablations show that the components are complementary and that directionality helps.
Critical weaknesses
  1. The core idea, learning a directed, goal-conditioned distance from offline trajectory connectivity and using it as a planning cost, is well established. Applying it to world models and CEM does not overcome any new challenges or provide additional insights.
  2. Weak experimental evaluation: the tasks are short-horizon standard benchmarks, and the qualitative examples are selected failure cases. The authors claim that Figure S1 shows cases where LeWM fails and TGCC succeeds, but the PushT example is clearly a failure, since the T's final position is even farther from the goal than its initial position.
Questions for authors
  1. How does the method perform on out-of-distribution (OOD) states or goals?
  2. What assumptions does graph construction make about the dataset? For example, trajectories presumably need to share overlapping or nearby states so that disjoint trajectories can be connected.
1 comment

I thank the authors for their hard work and the many detailed additional results.

On motivation and contribution. In my view, a robotics paper ultimately aims to complete the task. Taking PushT as a concrete example: if the final pose has a larger error than the initial pose, I would not consider the episode a success. If the paper instead targets a different problem, such as feature prediction rather than trajectory planning, then this task is not an appropriate way to demonstrate its performance. Comparisons with baselines cannot resolve this concern.

On novelty and generalization. I now better understand the problem the authors aim to solve: distilling global, demonstration-derived information into a local, step-wise planning cost through a separate network, without modifying the encoder. However, this raises a concern about generalization. I am less concerned with visual shifts (color, shape) than with layout shifts. The world model learns local one-step dynamics, which can transfer to new layouts, whereas hηh_\eta distills global multi-step reachability from the specific training trajectories. Table 3 (Spearman 0.949) and Table 2 (99.2% on Two-Room using hηh_\eta alone) suggest that it has memorized the topology of the training layout. I would therefore expect it to mislead the planner when the layout changes, for example, when the doorway in Two-Room is moved while hηh_\eta remains trained on the original map. Rebuilding the graph would require data from the new layout, which undermines the zero-shot planning motivation of JEPA world models. Could the authors evaluate this setting?

Reviewer_cdpoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the problem that the terminal latent similarity in world models mismatches with the planning target. Thus, the author proposes TGGC, which is a lightweight framework to add a learned control score to latent world model planners. During inference, the rollout score is combined with terminal latent similarity in MPC. Experiments show the performance of the proposed framework.

Critical strengths

The framework is really lightweight, and the performance increase is clear across tasks. Also, the diagnosis of the native LeWM latent distance with obstacle-aware path cost is helpful for understanding the motivation and the performance of this design.

Critical weaknesses
  1. Learning goal-conditioned distances or scores from offline data and using them as planning signals has been studied before, e.g., TD-MPC, CMD temporal distance. Thus, the novelty of this method is not clear, and the paper does not show whether the trajectory graph is needed compared with learned distances.
  2. The graph and cost estimator are environment-specific and trained with the same distribution as the evaluation tasks. It is unclear if this design can generalize to new cases or environments. The lack of a generalization experiment is very important for understanding the design.
Questions for authors
  1. Can the author provide some experiments to study how well this framework can handle generalization tasks?
  2. How does TGCC compare with using a learned temporal distance as the control score in the same planner?
Reviewer_6zbiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that distances in the latent representations learned by JEPA-style world models may not be suitable objectives for Model Predictive Control (MPC). The authors propose Trajectory-Graph Control Calibration (TGCC), a lightweight post-hoc method that keeps the world model frozen and learns a directed control-cost estimator from a graph of offline trajectories. During planning, this learned score is combined with the model's original latent distance to rank candidate action sequences. Experiments on Two-Room, Reacher, PushT, and Cube with LeWM and PLDM backbones show overall improvements in planning performance with limited additional overhead.

Critical strengths

The problem is clearly defined. Fig. 1 illustrates the mismatch between LeWM's original latent-distance cost and the ground-truth remaining path cost in Two-Room. TGCC is lightweight and can be added to an existing world model without retraining its encoder or predictor. The paper reports an additional parameter count equal to 4.19% of LeWM and a 2.4–3.1% increase in full CEM-solve latency. The paper reports cases where the method provides no improvement. For example, TGCC does not improve the reported PushT success rate of PLDM.

Critical weaknesses

Novelty: The contribution relative to prior work needs clearer discussion. Although the paper discusses graph-based planning, connections to Quasimetric RL (Wang et al., 2023), Contrastive RL (Eysenbach et al., 2022), and temporal-distance learning (Myers et al., 2024) are missing. Destrade et al. (arXiv:2601.00844) study aligning JEPA representation distances with goal-conditioned value for MPC and are closely related to the paper’s motivation. The paper should clarify what TGCC adds beyond these approaches and better justify its “first framework” claim. Recent preprints, including Traj-LeWM (arXiv:2608.14125), World Model Control by Trajectory Reachability Metrics (arXiv:2605.22164), and Temporal-Distance JEPA (arXiv:2607.25337), also appear relevant. I encourage discussing them in a revision, but do not count their omission against the paper.

Baselines: Table 1 compares TGCC against different agents and planners, but does not compare alternative learned control costs using the same frozen representations and CEM planner. Table S8 already isolates the effect of stitching, but does not answer this broader comparison. A controlled comparison that also matches the offline data and planning budget would help establish whether the gains depend on the proposed graph-based supervision or could also be achieved with other learned goal-reaching costs.

Results: The PLDM results need more discussion. The Reacher improvement is supported by the seven-seed results in Appendix F.5. However, TGCC does not improve the reported PushT success rate, and the gains on Two-Room and Cube are reported without standard deviations. Multi-seed results on these tasks, similar to those provided for Reacher, would help establish whether the improvements are consistent across tasks. The paper should also provide a task-specific discussion of why TGCC does not improve PLDM on PushT.

Questions for authors

How is the "w/o Graph Directionality" ablation implemented? What changes are made to the graph, the training labels, or the scoring network? Appendix C distinguishes the headline benchmark results from the separate multi-seed robustness evaluation. For Table 1 specifically, how many seeds and evaluation episodes were used for LeWM and the TGCC variants? Which baseline results, if any, were taken from prior work, and which were reproduced by the authors? Appendix B reports that no accepted stitching edge directly crosses the separating wall in Two-Room. Were corresponding checks performed on Reacher, PushT, and Cube to assess whether the accepted latent-neighbor matches justify bidirectional connections?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.