acceptodds
Under review as a conference paper at ICLR 2027

Low-Dimensional Decision Spaces for Latent World-Model Planning

Abstract

Latent world models can make visual planning more efficient by predicting in representation space rather than pixel space. However, conventional methods still predict a separate high-dimensional latent trajectory for every candidate action sequence, even though much visual information is shared across candidates. We study the variation among candidate outcomes for a fixed context, goal, and planning horizon, and refer to it as the *decision space*. Our analysis shows that a small number of directions captures most candidate variation while preserving much of the candidate-ranking signal used for action selection. Based on this finding, we introduce the Low-Dimensional Decision World Model (LDD-WM). LDD-WM performs one shared, action-free visual rollout and uses an action-conditioned component to update a compact -dimensional state for each candidate. The model is trained from physical-cost ordering and avoids reconstructing a separate full latent trajectory for every candidate. Across six datasets, LDD-WM consistently outperforms conventional full-latent reconstruction world models while requiring less computation and lower latency at inference. On MetaWorld Reach-Wall, LDD-WM improves success from to over JEPA-WM while reducing inference computation by a factor of and running faster. These results support decision-space modeling as an efficient alternative to full-latent prediction for visual planning.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Visual Planning, World Models, Representation Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_t7twOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that candidate-dependent variation in latent world-model predictions occupies a much smaller space than the full visual representation. It first analyzes PCA of predicted endpoints and the correlation between reduced latent scores and physical costs. It then introduces LDD-WM, which computes one shared action-free visual rollout and maintains a compact action-conditioned state for each candidate. A learned scoring function is trained with pairwise physical-cost ordering rather than candidate-specific latent reconstruction. The paper reports substantial planning improvements and approximately 68-71x lower latency than JEPA-WM in simulation, together with real-image navigation results.

Critical strengths
  1. The computational decomposition is compelling. Sharing expensive visual prediction across candidates addresses a concrete source of redundant computation. The decomposition of squared latent distance into a shared term, a goal-interaction term, and candidate response energy is algebraically clear.

  2. The reported efficiency gains are large and accompanied by control results. On Reach-Wall, Table 2 reports 64.6% success versus 29.9% for JEPA-WM, with planning-call latency falling from 14.86 s to 0.21 s. Results include three-seed variation rather than only single runs.

  3. Several important controls are already present. JEPA-WM-CF controls for additional counterfactual outcomes, DIRECT-SCORE uses matched ordering supervision, and the Full-Dimensional baseline in Figure 3 uses the ordering objective. These are stronger controls than comparing only against reconstruction-trained models.

  4. The analysis distinguishes variance from ranking information. Section 3.2 explicitly notes that a low-energy direction can still matter through its interaction with the goal. Reporting score-cost correlation alongside captured PCA energy is therefore appropriate.

Critical weaknesses
  1. Preserving score correlation does not establish reliable action selection. Section 3.2 and Table 1 analyze context-specific PCA under a fixed candidate count, horizon, and proposal distribution. The observed compression is meaningful, but retaining Spearman correlation with physical cost does not guarantee retaining the best candidates or low decision regret; the full-score correlation on Wall is itself only 0.321. The connection to planning would be stronger with top-candidate quality or regret measurements, together with tests under longer horizons and changed proposal distributions.

  2. The contribution of the shared visual rollout is not isolated. DIRECT-SCORE and the ordering-trained Full-Dimensional baseline are valuable controls, but neither isolates the benefit of predicting a shared visual trajectory from that of using a lightweight, shared-context candidate scorer. Replacing the reference rollout with repeated current features while retaining the same compact dynamics and scoring heads would provide a more targeted comparison. Moreover, the independently learned interaction and energy terms in Eq. (21) need not correspond to one latent response as in Eq. (10); the geometric decomposition motivates the scoring architecture but does not establish its necessity.

  3. The dependence on physical-cost supervision limits the demonstrated applicability. Sections 4.5 and 5.1 train candidate ordering using outcome costs, with training candidates drawn from the planning proposal family. This is a stronger supervision requirement than learning latent dynamics from observation-action trajectories alone. JEPA-WM-CF and DIRECT-SCORE appropriately control for data and supervision in the reported comparisons, but the study does not establish how the method behaves when candidate-cost labels are sparse or noisy, or when deployment proposals differ from training. The large inference savings should therefore be interpreted alongside this supervision requirement, rather than as an unconditional replacement for reconstruction-based world models.

  4. Real-image prediction results provide limited evidence of closed-loop navigation capability. Table 3 defines success by the predicted trajectory's final position error, rather than by executing a policy in the physical environment. Although LDD-WM improves the reported trajectory metrics, its success on TartanDrive remains only 2.0%. These results support improved offline trajectory prediction, but do not establish reliable real-world navigation. An executed closed-loop evaluation would better test whether compact scoring remains effective as action errors and observation shifts accumulate.

Questions for authors
  1. Does the compact model preserve top-candidate quality and low decision regret under longer horizons and proposal distributions different from those used in training?
  2. How much performance changes when the shared reference rollout is replaced with repeated current features, keeping the compact candidate dynamics and scoring heads fixed?
  3. How sensitive is LDD-WM to the number and accuracy of candidate-cost labels? Does its advantage persist with a restricted supervision budget?
  4. Do the real-image trajectory improvements translate into executed closed-loop navigation success, particularly in the challenging TartanDrive setting or a comparable environment?
Reviewer_Uy6jOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This study proposes LDD-WM. Based on the finding that futures varied by multiple action candidates can actually be separated into a shared visual reference and low-dimensional variation, LDD-WM trains a model capable of low-dimensional ranking. To achieve this, it trains an action-free world model that predicts the shared visual reference and a scoring model that considers the context, action sequence, and goal. By requiring only a single prediction of the shared visual reference, it demonstrates a powerful acceleration compared to baselines and shows results where the success rate matches or improves.

Critical strengths

It seems like a sound idea that most action-dependent candidates share a common visual reference, and that the actual differences can be represented in a lower-dimensional space, which is all we need for ranking.

Critical weaknesses
  1. The necessity of the proposed structure has not been sufficiently proven.
  • I acknowledge that there are many components and they align with motivation, but I do not fully understand their necessity. It is unclear if they are learning exactly as intended. For example, is it necessary to train with the score decomposed? Can't it be trained using an MLP(context, goal, action) while preserving other components of the model? Are they really aligned with the decomposition in the motivation section? Also, theoretically, zrefz_{ref} can be anything, so why is an action-free WM required? In my opinion, the baselines presented in the paper were not sufficiently controlled to answer these questions.
  1. The positioning of the work is unclear and this makes difficult to understand the work.
  • It is difficult to determine whether the evaluation setup performed well because a truly good world model was created, or because a scorer was trained. Also, unlike world model baselines, it learns to rank action candidates in advance; wouldn't this naturally lead to better performance on tasks? Additionally, navigation experiment does not seem to be a setup that requires a world model; since it simply involves following expert trajectories, it appears to be a problem that could potentially be solved with BC, yet there is no actual comparison with controlled version of BC trained on the same data. These points raise the question of whether this method is closer to a learned critic model or implicit BC rather than a world model. From this perspective, related work and baselines need to be reinforced, and the positioning needs to be clarified.
Questions for authors
  • I understand that action-free WM roughly predicts the average future. While this seems reasonable for manipulations viewed from a fixed view where the background doesn't change, I am curious why it works in navigation scenarios where scenes change rapidly. It would also be helpful to see an actual visualization of action-free WM.

  • Theoretically, zrefz_{ref} can be anything, but I do not understand the necessity of action-free WM. Please demonstrate the necessity of action-free WM by comparing it to performing the process directly on the initial image.

  • Can you explain whether each component of the decomposed learned score is related to each component developed from the motivation? Can you show that they must be learned by decomposing them in this specific form? What are the reasons why an MLP(context, goal, action) cannot be used as a scorer for your model?

  • It is difficult to determine whether the evaluation results were successful because a truly good world model was trained, or because the scorer was trained. Also, unlike world model baselines, it learns to rank candidates in advance; wouldn't this naturally lead to better performance on tasks? Is it possible to clarify these points?

  • The navigation experiment (NWM protocol) does not seem to be a setup that requires a world model. Since the goal is a future frame of the same trajectory and the evaluation is based on the distance from the recorded path, it could be solved through BC that follows the recorded path, and LDD-WM appears to simply perform implicit BC. How does LDD-WM differ from BC, and could you compare this with a controlled BC baseline? Also, could you show LDD-WM’s merits on more complex tasks with several environmental factors such as collision?

Reviewer_MbpzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents LDD-WM, a low-dimensional decision space for world model. The intuition is that ranking candidate actions is relative-based; therefore, LDD-WM moves ranking in high-dimensional space to low-dimensional PCA decomposition. Experiments on navigation and reach benchmarks show reduction in speed and better performance than ranking in original space.

Critical strengths
  • The motivation is well-presented and the qualitative toy experiment (Figure 1) clearly demonstrates the idea.

  • LDD-WM significantly reduces the inference speed over vanilla ranking both intuitively and empirically.

  • The method clearly shows improvements over full-latent reconstruction WM baselines.

Critical weaknesses
  • I find there is a disconnect from the math to neural network design. psi_h, a latent feature extracted from visual observation, serves as a r latent-channel vector to reduce the dimension in implementation (Sec 4.4); but in Sec. 3.2, this is PCA decomposition. This mismatch raises a question about how good psi_h can approximate variations in the original high-dimensional decision space. Why this is important because Table 1 runs diagnostic for PCA directions, therefore, it wouldn’t serve as a strong evidence to the network design if this disconnection is severe.

  • Experiments are thin and do not adequately support the low-dimensional solution. The results focus only on navigation and reaching benchmarks, which, I believe the action space can be naturally compressed in low dimension. It is unclear whether low-dimensional solution still works for contact-rich tasks. I suggest evaluating LDD-WM on contact-rich tasks, e.g., PushT (in Dino-WM) and other MetaWorld’s pick-place tasks.

  • The reduced low-dimensional r feels like a hyperparameter than a critical design choice. Tables 4 in supp. shows that Spearman correlation saturates from r=16 but the success rates do not stabilize from r=16 (~10% drop in PointMaze from r=16 to r=64). This result suggests the low-dimensional reduction seems to learn task-centric adaptation rather than capturing variation; and the model just works best at r=16.

  • Baseline comparison is also not fully satisfied. The efficiency gain is mainly over full-latent reconstruction baselines, it is highly recommended to include empirical comparisons with other efficient methods, such as Sparse Imagination and LeWorldModel [1], working on the similar benchmarks.

  • Ablation studies are missing. It is unclear how much of the improvement comes from ranking supervision with physical costs (J in Eq. 22) rather than from the low-dimensional compression. Furthermore, L_ord in Eq. 24 is not also not ablated and the motivation behind using margin loss is completely absent.

  • Minor comment: L174 text got truncated due to a possible vspace command. Please make sure the paper’s spacing follows ICLR format.

References:

[1] Maes et al. LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels.

Questions for authors
  • What is the evidence to support that the feature psi_h is a good approximation of v_i? Is LDD-WM simply an efficient design that reduces the latent dimension D?

  • Why Spearman correlation in Wall benchmark much lower than than Reach and Reach-Wall benchmarks (0.321 vs. >0.7) but still improve the performance?

  • Why the rerun ATE/RPE of NWM is higher than original number (1.179 vs. 1.13) on ReCoN? Furthermore, why is the rerun metrics match exactly the Forward model (5.752 vs. 5.75) on TartanDrive? NWM reports 1.13 on ReCoN and 5.63 on TartanDrive.

  • Can the authors conduct more experiments (include new benchmarks and baselines) to support the variation capture claim?

Reviewer_uR1pOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper follows the line of world-model planning exemplified by DINO-WM, which uses dense features from pretrained visual backbones. It proposes updating a low-dimensional state for each candidate action sequence while sharing a high-dimensional visual rollout across candidates. This reduces the computation required to evaluate different sampled action sequences. The model uses physical-cost-based ranking supervision to learn candidate scores.

The paper first decomposes the squared endpoint-to-goal latent distance into a shared term, an inner product between two residuals, and a squared-residual-norm term. It then applies PCA to identify the principal directions of candidate variation and empirically shows that a small number of directions can preserve much of the correlation between latent scores and physical costs.

Motivated by these empirical findings, the authors develop an architecture that uses a context-dependent direction matrix learned through ranking supervision, rather than explicitly computing a PCA basis.

Critical strengths

The architecture separates shared visual prediction from compact candidate-dependent dynamics. This allows high-dimensional visual information to be reused across candidate action sequences, reducing the need to predict dense features separately for every candidate while retaining goal-conditioned candidate scoring.

The reported results show substantial inference-efficiency gains. Under the settings in Table 2, LDD-WM reduces planning-call latency by approximately 68–71× relative to JEPA-WM while achieving higher mean success rates on the evaluated simulation tasks. These results provide empirical support for the practical value of sharing visual prediction and maintaining compact candidate states.

Critical weaknesses

Poor writing. Around lines 122–127, the quantities being defined should be placed on the left-hand side of the equations, as in Eq. (10), to make the definitions clearer. The choice and construction of zrefz^{\mathrm{ref}}, a central variable in the paper, should also be explained earlier. More generally, the connection between the method and the preceding analysis is weak.

Insufficient experimental coverage. The control evaluation primarily focuses on navigation and reaching tasks, leaving the applicability of the proposed low-dimensional decision representation to more complex object interactions insufficiently explored. Additional tasks such as PushT and Rope, both considered in DINO-WM, would help assess whether the findings extend to contact-rich manipulation and deformable-object dynamics. LeWorldModel should also be included as a baseline, since it provides an alternative approach to efficient planning through compact latent representations. Such a comparison should make differences in encoder training, supervision, and planning budgets explicit.

Questions for authors

Several aspects of the paper require further clarification:

At line 136, the analysis uses the mean of the predicted candidate endpoint latents as the reference: zref=zμ=1K∑k=1Kz^t+Hk.z^{\mathrm{ref}}=z_\mu=\frac{1}{K}\sum_{k=1}^{K}\hat z_{t+H}^{k}. In contrast, Eq. (12) at line 258 obtains the reference from the trained predictor: (zhref,Xh)=Fref(Wh−1).(z_h^{\mathrm{ref}},X_h)=F_{\mathrm{ref}}(W_{h-1}). Changing a shared reference does not invalidate the distance decomposition or alter pairwise differences between candidate responses. However, how do the learned reference branch and compact candidate dynamics realize the representation motivated by the PCA analysis? Please clarify the relationship between the learned representations and the candidate-dependent variation examined in Section 3.2, and provide diagnostic evidence supporting this connection.

The analysis shows that retaining a small number of principal components preserves much of the correlation between latent scores and physical costs. However, this does not establish that action selection or final task success is preserved. This observation alone is therefore insufficient to justify the proposed design, and the connection requires further evidence.

Equation (18) uses two separate heads to process each candidate’s low-dimensional state: one predicts a vector used in the inner-product interaction term, and the other predicts a nonnegative response-energy term. In the preceding derivation, both terms are determined by the same residual and are therefore geometrically linked. Why are they predicted independently rather than constrained to follow this relationship? The main text does not provide an ablation to justify this choice.

How are the counterfactual candidate sets and their physical costs JkJ_k constructed, particularly for the real-image datasets? Offline trajectories generally do not contain the outcomes of arbitrary alternative action sequences from the same initial state. Please specify how these outcomes or their costs are obtained, what additional state, pose, or task-specific information is required, and whether this provides more training information than prior world-model methods trained only on observations and actions.

If additional physical information is used, please quantify it and isolate its contribution to performance. The existing JEPA-WM-CF and Full-Dimensional baselines provide useful partial controls. Building on these, please report a CEM success-rate ablation that distinguishes the effect of physical-cost supervision from the choice of candidate-scoring function. In particular, compare the learned ranking score with conventional L1 and/or MSE distances between predicted endpoint latents and goal latents, and include a condition trained without physical-cost labels. This comparison would clarify how much of the improvement comes from the proposed architecture and how much depends on additional physical-cost supervision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.