acceptodds
Under review as a conference paper at ICLR 2027

Latent World Models Should Plan with Distances Along the Latent Manifold

Abstract

Latent world models such as LeWM plan by searching for the action sequence whose predicted terminal latent is closest to the goal latent under a distance measured in the latent space, such as the Euclidean distance. Planning with such distances assumes that proximity in the latent space measures how far the state is from the goal. For LeWM, we find that this assumption holds only in a small neighborhood near the goal, while the cost forms spurious low-cost basins that trap the planner. Yet neighborhood-based embeddings show that the latents retain the physical state. We hypothesize that the latents lie near a curved manifold along which proximity differs from proximity in the ambient space. If so, the terminal cost should measure the distance along this latent manifold. We propose the Neighborhood Graph Cost (NGC) to estimate this distance by the shortest path on a mutual nearest-neighbor graph over the latents of training observations. NGC changes only the terminal cost and requires no retraining. Experimental results show that NGC improves the success rate of LeWM on all four tasks, with the gain growing as the goal moves farther from the start. The results support the manifold hypothesis and our argument that latent world models should plan with distances along the latent manifold.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Latent world model, joint-embedding predictive architecture, model-based planning, latent space geometry

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_fR7JOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper investigates the limitations of Euclidean distance as a terminal cost for goal-conditioned planning with latent world models and proposes Neighborhood Graph Cost (NGC) as an alternative. The authors observe that Euclidean distances in LeWM representations are informative only within local neighborhoods and can produce misleading low-cost regions for distant goals, despite the latent representations retaining task-relevant state information. To address this issue, NGC constructs a mutual nearest-neighbor graph from encoded training observations and estimates distances between predicted terminal states and goals using shortest paths along the graph. The method replaces only the planning cost and requires no additional model training. Experiments across four continuous-control tasks demonstrate substantial improvements over Euclidean latent distance, particularly for longer goal horizons. Additional ablations, oracle-predictor evaluations, computational comparisons, and experiments with alternative checkpoints and representations support the effectiveness of the approach while highlighting its dependence on neighborhood quality.

Critical strengths
  1. The paper presents a well-motivated approach to addressing the mismatch between Euclidean latent distance and task-relevant progress in goal-conditioned planning. By replacing only the terminal cost with a graph-based distance, NGC achieves substantial improvements across four continuous-control tasks while keeping the world model and planner unchanged. The gains are particularly significant at longer goal offsets, where Euclidean distance becomes less informative. The evaluation using released LeWM checkpoints further supports the robustness of the main findings. The simplicity of the proposed intervention and its strong empirical performance make this a practically meaningful contribution.

  2. The paper provides several useful experiments beyond the main performance comparisons. In particular, the shortest-path ablation demonstrates that the improvements primarily arise from graph-based path distances rather than local density corrections or nearest-neighbor attachment alone. The oracle-predictor experiments help distinguish the limitations of the terminal cost from world model prediction errors, while the CEM budget comparison shows that simply increasing the search budget does not eliminate the performance gap. The authors also investigate alternative representations, graph sensitivity, and failure cases in contact-rich environments. These analyses provide relatively strong evidence for the proposed explanation while acknowledging its practical limitations.

Critical weaknesses
  1. Although the paper reports relatively modest online planning overhead, NGC requires constructing a neighborhood graph and precomputing all-pairs shortest-path distances, which introduces additional offline computation and quadratic storage complexity. Appendix H reports approximately 100 MB for the distance table with 5,000 graph nodes, but the preprocessing time, peak memory consumption, and scaling behavior for substantially larger graphs are not fully evaluated. Since a sufficiently dense graph may be necessary to preserve local manifold structure in complex environments, understanding these costs is important for assessing the practicality of NGC beyond the relatively small graphs considered in the current experiments. A more comprehensive scalability analysis would strengthen the paper's claims regarding computational efficiency.

  2. The proposed method assumes that nearby latent representations correspond to meaningful local relationships in the underlying environment, but this property is not guaranteed across different world model architectures or data distributions. The LpWM experiments provide useful evidence of this dependence, showing that NGC yields limited improvements on Cube and PushT when local neighborhoods are less faithful to physical state geometry. Moreover, the graph-size and neighborhood-size ablations reveal substantial performance degradation under insufficient coverage or overly large neighborhoods, particularly at longer goal offsets. Although the paper acknowledges these limitations, its evidence remains concentrated on closely related latent world model representations. Further validation on different representation families would help establish when the manifold-based cost is reliable and when graph construction may introduce misleading connections.

Questions for authors
  1. How does the computational cost of NGC scale with the number of graph nodes? Appendix H reports the storage requirements of the precomputed shortest-path table and relatively modest online latency at the evaluated graph sizes. Could the authors extend the graph-size experiments to report offline graph construction time, all-pairs shortest-path preprocessing time, peak memory consumption, and online planning latency as the number of nodes increases? Since the method relies on a quadratic distance table, understanding its computational behavior at larger scales would materially affect my assessment of its practical applicability.

  2. How well does NGC generalize to world models with different representation-learning strategies? The additional LpWM experiments provide valuable evidence that NGC's effectiveness depends on the fidelity of local latent neighborhoods. However, LeWM and LpWM remain relatively closely related representation-learning frameworks. Could the authors evaluate NGC on a world model using frozen pretrained visual representations, such as DINO-WM, while keeping the model and planning budget fixed? Reporting both L2 and NGC performance together with local neighborhood-quality measurements would help clarify whether the proposed approach generalizes beyond the evaluated representations and identify the conditions under which graph-based distances are reliable.

4 comments
AuthorsOpenReview ↗

Response to Reviewer fR7J (Part 1/3)

We appreciate your constructive review and valuable suggestions. Below is our detailed response to your questions and comments.

W1 / Q1: How does the computational cost of NGC scale with the number of graph nodes? Could the authors extend the graph-size experiments to report offline graph construction time, all-pairs shortest-path preprocessing time, peak memory consumption, and online planning latency as the number of nodes increases?

Thank you for raising this question regarding the practical applicability of NGC. We extended the graph-size experiments on TwoRoom and Cube to report all four requested costs:

CostTheoretical complexityTwoRoom N=625N=625Cube N=625N=625TwoRoom N=1250N=1250Cube N=1250N=1250TwoRoom N=2500N=2500Cube N=2500N=2500TwoRoom N=5000N=5000Cube N=5000N=5000TwoRoom N=10000N=10000Cube N=10000N=10000
Offline graph construction time (s)O(N2log⁡N)O(N^2\log N)0.0470.0490.2170.2160.9090.8883.8694.00432.09037.134
All-pairs shortest-path preprocessing time (s)O(N2log⁡N)O(N^2\log N)0.1250.1400.5570.5902.3482.55710.89411.62847.89751.656
Peak preprocessing CPU memory (GB)Θ(N2)\Theta(N^2)1.2301.2141.2561.2501.3151.3091.6251.6163.0483.850
Online planning latency (ms)O(Nlog⁡N)O(N\log N)593.3593.6594.3593.6594.4593.7598.6600.8609.2609.2

Measurements use an A100 40GB GPU and one numerical CPU thread, with three graph draws and three repeats per graph. Graph construction starts from encoded nodes, memory is the preprocessing worker's peak CPU RSS, and online latency covers a complete CEM solve.

When considering how these costs affect practical use, we would like to invite you to distinguish between one-time offline preprocessing costs and online deployment costs. We believe the offline costs have little impact on practical applicability: although graph construction and APSP scale as O(N2log⁡N)O(N^2\log N), they are performed only once for a fixed representation and graph. Like model training, this work happens before deployment and adds only a tiny fraction to the time needed to prepare the model for use: at 10,000 nodes, the two steps together take only 80–90 seconds. Compared with the 5.37 hours required to train the corresponding TwoRoom model on an RTX 4090, the graph and APSP computation is only 0.41% of that recorded training time. Moreover, if preprocessing RAM becomes limiting at larger scales, blockwise computation and disk-backed storage can reduce the memory requirement at the expense of additional preparation time.

Online latency and memory have a more direct impact on practical use because they determine how efficiently planning can run on the deployment hardware. Fortunately, the online scoring work scales as O(Nlog⁡N)O(N\log N), rather than quadratically, and remains very small relative to world-model prediction: even at 10,000 nodes, a complete CEM solve takes approximately 609 ms, only 5.4% above L2 on both tasks.

Online memory does not need to scale quadratically either: rather than keeping the Θ(N2)\Theta(N^2) distance table in RAM or GPU memory during planning, we only need to pay a small setup cost whenever the goal changes to extract a goal-specific vector of length NN. Although the full FP32 table occupies 100 MB at 5,000 nodes and 400 MB at 10,000 nodes, planning toward a fixed goal only requires each node's distance to that goal through its anchors. When the goal changes, we read the distance rows for its four anchors from disk, add each anchor's distance to the goal, and take the minimum for each node. The result is a vector of length NN, occupying only 0.02 MB for 5,000 nodes and 0.04 MB for 10,000 nodes, so each candidate can be scored using its distances to its own anchors and the cached anchor-to-goal values. With the full table kept on disk, finding the goal anchors and constructing this vector took median times of 6.5 ms on TwoRoom and 9.2 ms on Cube at 10,000 nodes in a CPU microbenchmark on our shared filesystem (p95: 41 and 237 ms). This setup cost is paid only when the goal changes, while all subsequent planning calls reuse the vector and require only linear O(N)O(N) distance storage in memory. We will include these scaling results and discussion in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer fR7J (Part 2/3)

W2 / Q2:

The proposed method assumes that nearby latent representations correspond to meaningful local relationships in the underlying environment, but this property is not guaranteed across different world model architectures or data distributions.

Further validation on different representation families would help establish when the manifold-based cost is reliable and when graph construction may introduce misleading connections.

Could the authors evaluate NGC on a world model using frozen pretrained visual representations, such as DINO-WM, while keeping the model and planning budget fixed? Reporting both L2 and NGC performance together with local neighborhood-quality measurements would help clarify whether the proposed approach generalizes beyond the evaluated representations and identify the conditions under which graph-based distances are reliable.

Thank you for this insightful suggestion. We agree that NGC relies on meaningful local relationships in latent space, which are not guaranteed across representations or tasks. If physically distant states are mapped close together, NGC can inherit misleading connections rather than correct them, limiting its benefits over L2.

Following your suggestion, we evaluated NGC on the DINO-WM checkpoints released by Point-LeWM, which use frozen DINOv2 features and task-specific predictors trained on the four environments. We kept the model, evaluation pairs, and planning budget fixed between L2 and NGC. To further remove the influence of prediction errors, we also evaluated an oracle predictor at the longest offset. Entries below show L2 → NGC success rates (%):

TaskOffset 25Offset 50Longest offsetLongest offset, oracle
TwoRoom100 → 10096 → 9862 → 8056 → 100
Reacher94 → 100100 → 10094 → 10094 → 100
Cube48 → 5844 → 4450 → 5468 → 70
PushT74 → 7046 → 4612 → 1412 → 18

These results show that NGC remains effective on TwoRoom and Reacher with frozen DINOv2 representations, while its gains on Cube and PushT are limited. This pattern is similar to what we observe with LpWM.

We also appreciate your suggestion to report local neighborhood quality alongside planning performance. We quantify neighborhood quality using the mean operational physical distance to the 15 nearest neighbors in latent space. Lower values indicate more physically local neighborhoods. Entries below report mean distance / longest-offset oracle L2 → NGC success (%):

Task (distance unit)LeWM–OracleLpWM–OracleDINOv2–Oracle
TwoRoom (px)3.14 / 19 → 1003.27 / 60 → 1003.25 / 56 → 100
Reacher (rad)0.1284 / 80 → 1000.1356 / 94 → 1000.1757 / 94 → 100
Cube (cm)13.94 / 37 → 7318.66 / 34 → 3815.79 / 68 → 70
PushT (px)111.75 / 13 → 38140.96 / 8 → 6148.23 / 12 → 18

On Cube and PushT, the representations used by LpWM and DINOv2 indeed have poorer local neighborhood quality than LeWM, corresponding to their smaller improvements from NGC over L2.

Continued in the next comment with additional AntMaze results.

AuthorsOpenReview ↗

Response to Reviewer fR7J (Part 3/3)

Response to W2 / Q2 (continued)

To further examine whether NGC’s benefits beyond LeWM extend to more complex tasks, we additionally evaluated visual AntMaze from OGBench, where an eight-actuator quadruped must coordinate locomotion and repeated foot–ground contacts while navigating around walls. We did not find an available DINO-WM checkpoint for this task, and training one was beyond our rebuttal time budget. We therefore evaluated frozen DINOv2 features only with an oracle predictor.

RepresentationPredictorMean distance (m)Offset 25Offset 50Offset 100
LeWMLearned0.35358 → 10040 → 9844 → 90
LeWMOracle0.35356 → 10040 → 10040 → 100
LpWMLearned0.35686 → 10062 → 9452 → 92
LpWMOracle0.35686 → 10060 → 10046 → 100
DINOv2Oracle2.77878 → 9460 → 6646 → 52

Interestingly, unlike the pattern across the other four tasks, LpWM and DINOv2 behave quite differently on AntMaze. Both LpWM and LeWM have good neighborhood quality and benefit substantially from NGC, whereas DINOv2 has considerably poorer neighborhood quality and gains much less.

We acknowledge that NGC is not highly effective across all representations and tasks. Nevertheless, its benefits are not confined to the LeWM framework or to relatively simple tasks such as TwoRoom and Reacher. Understanding why different representation-learning strategies produce different neighborhood quality is a challenging and interesting question that we intend to investigate further. Although we do not yet fully understand the underlying causes, we believe that these results, showing how NGC’s effectiveness varies across representations and tasks, are valuable to share with the community. They may motivate further discoveries and methods that improve upon NGC. As suggested by our paper’s title, our central message is to offer a manifold perspective on planning with latent world models, with NGC serving as a concrete method for exploring this perspective.


Summary

Thank you for guiding us toward experiments that strengthen both the practical case for NGC and our understanding of when it helps. They show modest overhead at the tested scales and benefits beyond the originally evaluated representations, including frozen pretrained features. The extent of these benefits varies with the task and representation-learning strategy, raising intriguing questions about the underlying mechanisms. We will incorporate these results and their implications for planning into the final paper. Please let us know if any concerns remain or if further clarification would be helpful during the discussion.

Thank you for the additional experiments and clarifications. The scalability measurements clarify the practical costs at the evaluated graph sizes, while the DINO-WM and additional AntMaze experiments broaden the empirical support beyond the original settings. I also appreciate the neighborhood-quality analysis, which helps explain both the gains and the limitations across representations. My main concerns have been satisfactorily addressed. Given the paper’s useful insight, simple approach, and strengthened empirical evidence, I am raising my score from 3 to 4.

Reviewer_z1zbOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates why Euclidean terminal costs can misguide planning in latent world models even when their representations retain task-relevant state information. It proposes Neighborhood Graph Cost (NGC), which estimates distances using shortest paths on a mutual nearest-neighbor graph of encoded training observations. NGC keeps the world model and planner unchanged and requires no additional model training. Experiments across four control tasks demonstrate improved performance.

Critical strengths
  1. The paper is clearly written and easy to follow, with visual diagnostics that make the motivation intuitive. The experiment results, ablation studies and comparisons across different CEM search budgets provide convincing evidence that the proposed terminal cost contributes to the planning gains.

  2. The proposed NGC substantially improves long range planning across the four evaluated tasks while keeping the trained world model and CEM planner unchanged. It requires no additional model training, making the reported gains practically meaningful.

Critical weaknesses
  1. The comparison with other terminal cost alternatives is missing. Comparing against other parametric terminal costs under the same frozen world model and CEM budget will be helpful to clarify the benefits and trade-offs of NGC’s training-free cost construction.

  2. Long range performance is sensitive to graph construction. Table 13 shows that settings with similar short range performance can differ substantially at longer horizons, particularly on TwoRoom. This raises concerns about the robustness of the long range gains across graph configurations.

Questions for authors
  1. Under the same frozen checkpoints and CEM budget, how does NGC compare with other parametric terminal costs learned from the available trajectories?

  2. Given the sensitivity of long range performance to graph configuration, what practical guidance can the authors provide for selecting graph size and neighborhood size for reliable long range planning, particularly when short range validation does not distinguish configurations with different long range performance?

4 comments
AuthorsOpenReview ↗

Response to Reviewer z1zb (Part 1/3)

We appreciate your constructive review and valuable suggestions. Below is our detailed response to your questions and comments.

W1 / Q1: Under the same frozen checkpoints and CEM budget, how does NGC compare with other parametric terminal costs learned from the available trajectories?

Thank you for suggesting this comparison. We have added a comparison with TRM (Li et al., 2026), which learns a pairwise cost over latent states and uses it as the terminal cost for planning. We keep the frozen encoder and learned predictor, evaluation pairs, and CEM budget fixed across L2, TRM, and NGC within each task. Across all five tasks, we train the TRM cost head to predict the temporal separation between states from the same trajectory, without task-state labels or a hybrid cost.

To extend this comparison to a more complex control setting, we additionally include visual AntMaze from OGBench, where an eight-actuator quadruped must coordinate locomotion and repeated foot–ground contacts while navigating around walls. The table below compares planning success rates across five tasks, using 100 evaluation pairs per setting:

TaskOffsetL2 (%)TRM (%)NGC (%)
TwoRoom2592100100
TwoRoom5065100100
TwoRoom75249794
Cube25443659
Cube50443553
Cube100312950
Reacher25999599
Reacher509797100
Reacher1008197100
PushT25908788
PushT50434961
PushT100141828
AntMaze256286100
AntMaze50415895
AntMaze100425392
Mean (15 settings)—57.9369.1381.27

NGC outperforms TRM by 12.13 percentage points in mean success rate, with advantages of 18–23 percentage points on Cube and 37–39 percentage points on AntMaze at longer offsets. We will include this comparison in the revised paper.

AuthorsOpenReview ↗

Response to Reviewer z1zb (Part 2/3)

W2 / Q2: Given the sensitivity of long range performance to graph configuration, what practical guidance can the authors provide for selecting graph size and neighborhood size for reliable long range planning, particularly when short range validation does not distinguish configurations with different long range performance?

Thank you for highlighting this concern. We expanded the sensitivity study to five tasks and two world models, LeWM and LpWM. The table below reports LeWM / LpWM success rates (%) at the longest offset, with L2 as a reference. The default NGC configuration is shown in bold.

NNkkTwoRoomReacherCubePushTAntMaze
L2—24 / 7081 / 9631 / 3414 / 1244 / 52
6251555 / 77100 / 10045 / 3519 / 1180 / 86
1,2501563 / 98100 / 10043 / 3824 / 1292 / 88
2,5001572 / 96100 / 10052 / 3626 / 1094 / 90
5,0001594 / 97100 / 10050 / 3928 / 1290 / 92
10,0001597 / 96100 / 10051 / 3634 / 1192 / 88
5,000597 / 98100 / 10049 / 3523 / 1082 / 88
5,0001099 / 97100 / 10049 / 3530 / 1094 / 88
5,0003068 / 97100 / 10050 / 3425 / 1396 / 86
5,0006045 / 75100 / 10050 / 3727 / 1294 / 88

The strongest sensitivity occurs on TwoRoom with LeWM. Reducing NN from 5,000 to 625 lowers success from 94% to 55%, while increasing kk from 15 to 60 lowers it to 45%. LpWM is more stable on TwoRoom across the intermediate configurations, although it also degrades at N=625N=625 and k=60k=60. The other four tasks show smaller absolute changes, with Reacher remaining at 100% throughout. These results motivate the following practical guidance.

Graph size (NN). NGC estimates distances along the latent manifold. With an appropriate neighborhood size, sampling more nodes from the same covered region should improve this approximation through greater sampling density. Within the available preprocessing, memory, and planning budget, we recommend starting with a larger node set and then checking whether it can be reduced without sacrificing long-range success.

Neighborhood size (kk). Both overly large and overly small neighborhoods can weaken NGC. With a large kk, pairs that should be connected through a path along the manifold can instead become direct neighbors, whose edge costs are Euclidean distances in the ambient latent space. These connections can shortcut the manifold and weaken the benefit of NGC. With a small kk, the mutual-neighbor graph can fragment into more disconnected components, requiring additional inter-component edges. Greater reliance on these added connections can also compromise the graph's approximation of the manifold.

We therefore recommend looking for configurations with both few bridging edges (BB) and low latent edge-length variance (VV). A large BB suggests that kk is too small, leaving many disconnected components. A large VV can indicate that kk is too large, admitting distant latent nodes that should not be treated as local neighbors. We compute the population variance over all undirected graph edges, including added edges, counting each once. We normalize it by a fixed scale: the squared mean 15th-neighbor distance in the same task/model's 5,000-node reference bank. At N=5,000N=5,000, the table below reports BB / VV / long-range SR (%) for NGC, and SR only for L2.

kkTwoRoom LeWMTwoRoom LpWMPushT LeWMPushT LpWMAntMaze LeWMAntMaze LpWM
L2247014124452
566 / 0.040 / 9760 / 0.043 / 98261 / 0.051 / 23165 / 0.049 / 10158 / 0.023 / 8272 / 0.044 / 88
101 / 0.069 / 992 / 0.077 / 9710 / 0.058 / 3014 / 0.057 / 1032 / 0.034 / 943 / 0.083 / 88
150 / 0.098 / 940 / 0.109 / 971 / 0.062 / 283 / 0.064 / 1212 / 0.047 / 900 / 0.123 / 92
300 / 0.177 / 680 / 0.200 / 970 / 0.067 / 250 / 0.076 / 132 / 0.088 / 960 / 0.241 / 86
600 / 0.310 / 450 / 0.365 / 750 / 0.068 / 270 / 0.091 / 120 / 0.172 / 940 / 0.453 / 88

In TwoRoom, increasing kk from 15 to 60 more than triples VV and lowers success by 49 and 22 percentage points for LeWM and LpWM, respectively, even though no component-connecting edges are needed. For PushT–LeWM and AntMaze–LeWM, increasing kk from 5 to 10 reduces BB from 261 to 10 and from 158 to 32, respectively, while keeping VV relatively low and improving success by 7 and 12 percentage points. These results support using both diagnostics to narrow the search.

When task rollouts are available, we also recommend validating the candidate configurations on a held-out set at the target planning range before fixing the final choice.

AuthorsOpenReview ↗

Response to Reviewer z1zb (Part 3/3)

Summary

We appreciate your suggestions for a broader baseline comparison and clearer practical guidance. The resulting experiments show substantial gains over TRM, particularly on Cube and AntMaze, and highlight the complementary roles of graph connectivity and neighborhood locality. These findings help explain why neither smaller nor larger neighborhoods are always preferable, providing a clearer basis for graph selection. We will incorporate these results and recommendations into the revised paper. Please let us know if any concerns remain or if further clarification would be helpful during the discussion.

Thank you for the detailed experiments and thoughtful response. The clarifications have largely addressed my concerns. I appreciate the authors' efforts in addressing my questions and will maintain my positive score.

Reviewer_oQ8AOpenReview ↗
Rating
3
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary
  • shows that Euclidean distance in LeWM's latent space is meaningful locally but fails to capture long-range goal distances (that's very nice)
  • proposes Neighborhood Graph Cost: shortest-path distances over a mutual kNN graph of training latents, without retraining
  • improves long-horizon planning across four tasks, with a particularly large gain on TwoRoom
Critical strengths
  • clear diagnosis of spurious low-cost basins and why Euclidean planning fails
  • simple method with strong improvements, requiring no additional training (albeit offline bank)
  • good ablations show gains come from graph shortest paths rather than neighborhood scaling
  • convincing controls with oracle predictions, increased CEM budgets, and different checkpoints
  • gains grow with goal distance (very important for planning)
Critical weaknesses
  • graph-based geodesic estimation is well established (Isomap)... the methodological novelty is limited
  • the manifold hypothesis is supported empirically but not formally established
  • graph construction requires offline coverage of the latent space. Scalability to large, complex environments is unclear and this is the biggest practical issue IMO
  • the method assumes locally meaningful Euclidean neighborhoods, which may fail around contact and discontinuous dynamics but is quite OK overall IMO
  • improvements are much smaller on manipulation tasks
  • cross-model generalization is mixed: NGC offers little improvement on LpWM for Cube and PushT, why?
Questions for authors

Some simple questions that would help me

  • how sensitive is NGC to graph size, neighborhood size k, and offline data coverage?
  • does the method remain effective when the graph is built from trajectories with poor state-space coverage?
  • can the gains be reproduced on higher-dimensional environments with multiple independently moving objects?
  • how does NGC compare with simpler graph-based alternatives, such as temporal transition graphs or reachability-weighted shortest paths? What explains the limited improvement on LpWM for Cube and PushT? Can neighborhood fidelity predict when NGC will help?
3 comments
AuthorsOpenReview ↗

Response to Reviewer oQ8A (Part 1/3)

Thank you for your constructive review and valuable suggestions. Below is our detailed response to your questions.

Q1: how sensitive is NGC to graph size, neighborhood size k?

Thank you for this helpful question. We extended the sensitivity study to include visual AntMaze from OGBench, where a quadruped must coordinate locomotion while navigating around walls. This allows us to examine whether sensitivity to graph construction also arises in a more complex control setting. We evaluate both LeWM and LpWM alongside the original tasks.

Each entry gives LeWM / LpWM success rates (%) at the longest offset, with the default configuration in bold.

NNkkTwoRoomReacherCubePushTAntMaze
6251555 / 77100 / 10045 / 3519 / 1180 / 86
1,2501563 / 98100 / 10043 / 3824 / 1292 / 88
2,5001572 / 96100 / 10052 / 3626 / 1094 / 90
5,0001594 / 97100 / 10050 / 3928 / 1290 / 92
10,0001597 / 96100 / 10051 / 3634 / 1192 / 88
5,000597 / 98100 / 10049 / 3523 / 1082 / 88
5,0001099 / 97100 / 10049 / 3530 / 1094 / 88
5,0003068 / 97100 / 10050 / 3425 / 1396 / 86
5,0006045 / 75100 / 10050 / 3727 / 1294 / 88

Graph size NN. Smaller graphs can reduce success substantially on TwoRoom, while the other tasks show milder changes. Increasing the graph beyond the default size brings further gains in some settings, but the benefit depends on the task and representation.

Neighborhood size kk. TwoRoom favors smaller neighborhoods and degrades with large kk, especially for LeWM. The other tasks are less sensitive, and the best-performing neighborhood size varies across tasks and representations.

Encouragingly, AntMaze maintains high success across the tested configurations for both models, showing that more complex control does not necessarily entail greater graph sensitivity. The strongest sensitivity appears on TwoRoom at the longest offset, highlighting the value of validating configurations at the intended goal distances.

Q1 / Q2: how sensitive is NGC to offline data coverage? does the method remain effective when the graph is built from trajectories with poor state-space coverage?

Thank you for your question. We tested NGC with graphs built from trajectories with strongly imbalanced state-space coverage. Only 5% of the trajectories ever visit the right-hand region: the right room for the TwoRoom agent, the right half of Reacher's end-effector workspace, and the right half of the image for the object centers in Cube and PushT. We keep the frozen world models and evaluate the models at the longest offset. Each entry reports LeWM / LpWM success rates (%).

TaskL2Standard-coverage NGC5%-coverage NGC
TwoRoom24 / 7094 / 9746 / 95
Reacher81 / 96100 / 100100 / 100
Cube31 / 3450 / 3945 / 36
PushT14 / 1228 / 1232 / 15

NGC retains higher observed success rates than L2 in all eight task–model settings under this coverage imbalance. The largest change relative to standard coverage occurs for LeWM on TwoRoom, while the remaining settings stay close to their standard-coverage results.

Interestingly, this pattern aligns closely with the graph-size and neighborhood-size sensitivity results. Across both studies, the most pronounced sensitivity is concentrated in LeWM on TwoRoom, with smaller changes in the other task–model settings. We will include these results and discussion in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer oQ8A (Part 2/3)

Q3: can the gains be reproduced on higher-dimensional environments with multiple independently moving objects?

Thank you for this helpful question. First, we extended our evaluation to visual AntMaze, where an eight-actuator quadruped must coordinate locomotion and repeated foot–ground contacts while navigating around walls. We obtain the following success rates with learned predictors. Entries report L2 → NGC (%):

ModelOffset 25Offset 50Offset 100
LeWM58 → 10040 → 9844 → 90
LpWM86 → 10062 → 9452 → 92

These results show substantial gains for both models in a higher-dimensional and more complex environment.

Following your suggestion, we additionally constructed a two-Ant extension of visual AntMaze. Two Ants, colored red and blue, are independently controlled within the same maze, giving a joint 16-dimensional action space. Success requires both Ants to be within their respective goal regions.

We trained a joint LeWM from scratch on 10,000 trajectories and compared L2 and NGC on the same 50 evaluation pairs at offset 100. Following the oracle-predictor evaluation in our submission, we also used simulator rollouts in place of learned predictions, keeping the encoder, graph, and planning settings unchanged. This comparison tests whether NGC’s gains can be reproduced in the two-Ant environment when prediction errors are removed.

PredictorL2 successNGC success
Learned22%24%
Oracle46%88%

With oracle prediction, NGC improves success from 46% to 88%. This directly demonstrates that, with accurate predictions, NGC’s gains can be reproduced in a higher-dimensional environment with multiple independently controlled agents. With the current learned predictor, however, success changes only from 22% to 24%. Thus, the oracle experiment establishes the potential benefit of NGC in this setting, while realizing this benefit with a learned predictor remains an open challenge. We will include both results and this limitation in the revised paper.

Q4: how does NGC compare with simpler graph-based alternatives, such as temporal transition graphs or reachability-weighted shortest paths?

Thank you for this question. In our view, NGC is already quite simple, so it was not immediately clear to us how to construct a simpler alternative along the lines you suggested. We therefore implemented and evaluated two temporal graph variants as concrete interpretations of your suggestion.

First, we sampled every fifth training frame and connected consecutive samples only within each episode. This leaves episodes disconnected. The planner scores candidate action sequences using the temporal-graph distance between their predicted terminal states and the goal, assigning unreachable routes a fixed finite penalty plus attachment costs. Success is measured by executing the selected actions in the simulator, rather than automatically counting disconnected cases as failures. On TwoRoom at offsets 25/75, success was 4%/8%, versus 100%/96% for NGC.

To improve connectivity, we added bidirectional, Euclidean-weighted edges linking each node to its nearest neighbor from another episode.

TaskOffsetsNGC (%)Temporal + cross-episode 1-NN (%)
TwoRoom25 / 75100 / 9684 / 82
PushT25 / 10090 / 3672 / 50
Cube25 / 10060 / 5052 / 28
Reacher25 / 10098 / 10060 / 84

NGC outperforms the temporal baseline by 14.75 percentage points in mean success rate, with advantages observed in seven of eight settings. NGC also only uses 5,000 nodes while the temporal baseline uses approximately 175,000–451,000 nodes. Including full-frame encoding and graph preparation, the temporal baseline took 3.3–9.1× as long as NGC on the same hardware, with additional shortest-path computation when the goal changes.

These results concern this specific temporal baseline. We would welcome clarification of the suggested reachability-weighted construction.

AuthorsOpenReview ↗

Response to Reviewer oQ8A (Part 3/3)

Q4 / W6: cross-model generalization is mixed: NGC offers little improvement on LpWM for Cube and PushT, why? Can neighborhood fidelity predict when NGC will help?

Thank you for this insightful question. To further investigate NGC's cross-model generalization, we evaluated NGC on the DINO-WM checkpoints released by Point-LeWM, which use frozen DINOv2 features and task-specific predictors trained on the four environments. We kept the checkpoint, evaluation pairs, and planning budget fixed between L2 and NGC. We also extended the evaluation to visual AntMaze and included oracle predictors to examine performance without learned prediction error.

To quantify neighborhood fidelity, we use the mean operational physical distance to the 15 nearest neighbors in latent space, with smaller distances indicating more physically local neighborhoods. The table reports mean distance / L2 → NGC success (%) at the longest offset.

Task (distance unit)PredictorLeWMLpWMDINO-WM
TwoRoom (px)Learned5.44 / 29.6 → 99.95.31 / 64.2 → 99.03.25 / 62.0 → 80.0
TwoRoom (px)Oracle3.14 / 19.0 → 100.03.27 / 60.0 → 100.03.25 / 56.0 → 100.0
Reacher (rad)Learned0.1288 / 79.1 → 100.00.1361 / 93.0 → 100.00.1757 / 94.0 → 100.0
Reacher (rad)Oracle0.1284 / 80.0 → 100.00.1356 / 94.0 → 100.00.1757 / 94.0 → 100.0
Cube (cm)Learned13.77 / 31.1 → 50.018.52 / 31.4 → 33.115.79 / 50.0 → 54.0
Cube (cm)Oracle13.94 / 37.0 → 73.018.66 / 34.0 → 38.015.79 / 68.0 → 70.0
PushT (px)Learned115.36 / 12.7 → 26.3155.97 / 8.0 → 8.0148.23 / 12.0 → 14.0
PushT (px)Oracle111.75 / 13.0 → 38.0140.96 / 8.0 → 6.0148.23 / 12.0 → 18.0
AntMaze (m)Learned0.353 / 44.0 → 90.00.356 / 52.0 → 92.0–
AntMaze (m)Oracle0.353 / 40.0 → 100.00.356 / 46.0 → 100.02.778 / 46.0 → 52.0

The last column uses DINO-WM with learned prediction and frozen DINOv2 features with oracle prediction. No learned DINO-WM checkpoint was available for AntMaze.

These results support using neighborhood fidelity to predict when NGC will help. On Cube and PushT, LpWM and DINOv2 have larger mean neighborhood distances than LeWM and obtain much smaller NGC gains, even with oracle prediction. NGC relies on these local relationships to construct paths, so poorer neighborhood fidelity can limit the benefit of its graph-based cost. AntMaze provides a particularly clear contrast: LpWM has neighborhood quality comparable to LeWM and benefits substantially from NGC with both learned and oracle predictors, whereas DINOv2 has much poorer neighborhood quality and gains little. Mean distance therefore helps distinguish settings with substantial and limited benefits, although the exact gain also depends on prediction accuracy and baseline headroom.

We acknowledge that NGC is not highly effective across all representations and tasks. Nevertheless, its benefits are not confined to the LeWM framework or to relatively simple tasks such as TwoRoom and Reacher. Understanding why different representation-learning strategies produce different neighborhood fidelity is a challenging and interesting question that we intend to investigate further. Although we do not yet fully understand the underlying causes, we believe that these results, showing how NGC’s effectiveness varies across representations and tasks, are valuable to share with the community. They may motivate further discoveries and methods that improve upon NGC. As suggested by our paper’s title, our central message is to offer a manifold perspective on planning with latent world models, with NGC serving as a concrete method for exploring this perspective.


Summary

Thank you for encouraging us to examine both the scope of NGC's benefits and the conditions that support them. Your questions prompted us to examine graph sensitivity more thoroughly, compare neighborhood quality across representations, and extend our evaluation to two independently controlled Ants. Together, these experiments strengthen the empirical case for NGC and highlight the value of neighborhood fidelity for anticipating when it will improve planning. The two-Ant evaluation further demonstrates that NGC’s benefits can extend to multiple independently controlled agents, as shown by the oracle-predictor results. We appreciate how your feedback has helped us present both the contribution and its limitations more clearly. We will incorporate these findings into the revised paper. Please let us know if any concerns remain or if further clarification would be helpful during the discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.