acceptodds
Under review as a conference paper at ICLR 2027

A Linear Map Goes a Long Way: Learned Costs for Planning with Frozen World Models

Abstract

Planning with a visual world model requires a cost that compares predicted outcomes with a goal. Learned costs outperform raw latent distance, but it remains unclear how much of this benefit requires a flexible scoring network. We keep the encoder and dynamics fixed and train three costs to predict the elapsed time between logged observations under identical conditions: a linear map, a shared nonlinear map, and the joint scorer of prior work. In TwoRoom navigation, the linear cost raises planning success from 18% to 96%, and neither flexible cost adds a clear gain. The result holds for a second family of world models, and across rooms the linear cost also outperforms a cost built from decoded positions. Whitening the latent space or training the same map on shuffled targets recovers almost none of the improvement, indicating that the temporal targets are essential to it. In PushT block pushing, the flexible costs fit elapsed time better, even on episodes excluded from cost training, yet the joint scorer plans worse than the linear cost. The linear cost raises success from 36% to 44%, while the joint scorer reaches only 29%, and this gap persists after tuning each family and after symmetrizing the joint scorer. A comparably trained linear cost is therefore the baseline that learned planning costs should beat, and temporal loss alone is not a reliable criterion for choosing among them.

Primary area:
applications to robotics, autonomy, planning
Keywords:
world models, model-based planning, latent-space planning, learned cost functions, metric learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 5 reviews, above 27% of the others.
Reviewer_r8ufOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper presents different ways to learn cost functions between a latent goal state and an observed "state" in latent world models and MPC/CEM planning, as opposed to direct L2 comparison between these two. It introduces linear and non-linear costs for quadratic functions and tests it on toy problems.

Critical strengths

The paper IMO addresses the right angle in WM + MPC settings, namely the cost functions. Comparing latent states is probably one of the most critical current weaknesses in this setup.

Critical weaknesses

I would like to reassure the human co-authors of this paper that I did not use an LLM for my review and that thus their paper has not been uploaded to an LLM either. I see that as an advantage for the submitters, but it also comes with a downside: the paper needs to be understandable for a human.

PRESENTATION

The paper is very hard to understand since it is written as a collection of details. While technically there is structuring into abstract, introduction, method, experiments, the levels of abstraction seem to be identical in all of these section. All sections present the same (high) level of technical detail and so we jump from technical detail to technical detail all along. The paper reads like a technical report for a member of the team already very familiar with the work. For instance (I cannot cover all issues in this review),

  • the abstract and introduction are written like experimental sections, full of details and results.
  • the method section 3 does not even introduce the basic setting of latent WMs and MPC, it is supposed to be known.
  • the method section mixes actual method and details on the experiments, frequently mentioning which kind of loss and distance is tested on which kind of problem.

The paper also contains claims which are probably true but heavily depend on assumptions which have never been specified. For instance, L075 "in navtio .... latent space suffices ...": this really depends on the kind of navigation problem addressed: what is input, what is output, is there a map or is this an agent in a 3D photorealistic environment?

There is lot of heavy language written in a highly detailed and non-standard way. Sentences with "deep" explanations are dropped in without context and I have a hard time understanding what is meant. Example: L190 "Training and validation pairs come from the same pool of source episodes, so validation loss measures fit within that pool." The sentence is understandable, but I don't understand what the authors want to say with this. This is clearly not really a desired behavior as the experimental setup does not require any OOD generalization, and "validation loss measures fit within that pool" is understandable but is not linked to any scientific motivation I could imagine.

It is a pity that the authors did not find a way to make use of the almost entirely empty last page.

NOVELTY

The setup tested in this paper is not novel. Learning temporal distances for goal reaching has been done in a large body of work, as also mentioned in the RW section. Comparing linear and non-linear costs has been done in the cited references (Li et al., 2026a) and Singh (2026). As mentioned, the contribution is the positive semidefinite quadratic cost. The conclusions however, stay very similar to the conclusions of prior work.

Section 3 is also written in way mixing contributions and existing work, it is not easy to understand what the authors claim.

What is the motivation of the factorized linear version? In the best case it is a low-rank linear variant with effect on optimization. The choice is not properly motivated.

The notation is hard to follow. why is there a d in from of unnormalized sum and no sum symbol (L145)? What is the s factor in L161? Why was a bias term introduced in L153 if it cancels out in equation (2)? While in L171 it is clear that \Sigma is the cov matrix, this should be introduced.

The experimental setup is not clear (section 3.3)

EXPERIMENTS AND RESULTS

The experiments are in line with the very simplistic experiments of the WM literature. While I do understand that sometimes overly simplistic environments (actually, "toy problems") can be justified to specifically test some particular line of promising models if a breakthrough is expected later, I think the community should rethink it's current view on world models. These kind of models are currently held in high esteem, which entirely contrasts with the toy problems they are capable of solving. The two environments covered in this work are good examples for this.

Beyond this remark, one has to ask the question what the take-home message of this work is, what did we learn from it? Other work has already provided evidence that linear cost functions can outperform non-linear ones in certain settings. However, what is the reason for this? In what settings can we believe this to hold? One of the most important (and bold) statements in this paper is "The frozen representation therefore supports a useful planning geometry, which temporal training of a linear cost recovers in these tasks. The linear cost also has practical structural advantages", L423. However, this is a universal statement and it potentially (and probably) depends on what "frozen" means, and what the task setup is, even for "navigation" tasks: is there a map, is there a state / position input, does it generalize to static point goal from photorealistic simulators or even to image goal? Generally speaking, which properties of a tasks do we rely on for this to hold?

SUMMARY

I will be upfront: the rebuttal is unlikely to change my opinion. My assessment is that the presentation alone makes the paper unsuitable for publication. Additionally, the experiments are performed on very simplistic tasks (which admittedly are somewhat standard in the WM literature) and the novelty is not high enough.

DETAILS

  • L136 a_t should be a sequence of actions if it predicts z_T+H(a_t)
Questions for authors

See "weaknesses" section.

Additionally:

  • Are training and validation split into different environments? Is there a test set?
  • What is "Development success"? (Figure 3)
  • What is "it in L351?
Reviewer_xEQBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper tries to explore the role of A LINEAR MAP in world model.

Critical strengths

The experimental improvements are substantial. Compared with using raw latent-space distance as the cost function, temporal supervision produces clear performance gains. This demonstrates that temporal distance is a reasonably suitable supervision signal compared with directly computing raw latent-space distance.

The ablation studies are relatively rigorous. The experiments using covariance whitening and shuffled temporal supervision validate the effectiveness of temporal supervision while ruling out the influence of differences in variability across latent representation dimensions.

Critical weaknesses

The generalizability of the conclusion that a linear cost function is sufficient requires further examination. The task environments are relatively simple and essentially consist of navigation and contact tasks on a single two-dimensional plane. They do not establish whether a linear cost function would retain an advantage over more complex and flexible cost functions in more complex or diverse environments. The apparent advantage of the linear cost function may reflect overfitting to the limited variety of environments.

There are shortcomings in the experimental setup. Results for the PushT task are reported with only one world model, and the number of planner seeds differs across the three experimental settings. These choices introduce inconsistencies in the experimental setup.

Some experimental observations lack in-depth explanations. For example, the paper does not adequately explain why the raw-distance visualization in Figure 4 contains a relatively concentrated low-cost region near the doorway. It also does not examine in sufficient depth whether the linear cost function's highest success rate despite its highest loss arises because nonlinear cost functions are more prone to overfitting.

Questions for authors

In the first panel of Figure 4, the raw latent-space distance in the TwoRoom task is unusually small near the doorway. This is puzzling because the distance is computed between visually encoded states in latent space. Why does this particular region near the doorway exhibit such concentrated similarity to the goal? The paper states that the visual encoder is frozen and should be task-agnostic, so it is unclear why a specific region far from the goal would exhibit such high similarity. Does this phenomenon persist across multiple goal locations? Please provide a convincing analysis of this observation.

Were the data used to train the cost function cleaned or filtered? Using temporal distance as a supervision signal depends on trajectory quality. If the trajectories contain substantial exploratory trial and error, this supervision signal could introduce considerable errors.

Why was the doorway location not randomized in the TwoRoom task? A fixed doorway causes trajectories between rooms to share the same passage bottleneck. The current experiments cannot distinguish whether the linear cost function recovers transferable planning structure or primarily adapts to a single fixed layout. Could comparisons across different doorway locations or layouts be added?

Reviewer_VMG4OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies planning costs for frozen visual world models. The authors compare three learned cost functions under a unified temporal-distance regression objective: a linear map, a nonlinear shared embedding, and a joint scorer. Experiments on TwoRoom navigation and PushT show that a simple linear cost often matches or outperforms more expressive alternatives, despite having lower temporal prediction accuracy in some cases.

Critical strengths
  1. The paper performs a relatively controlled comparison among linear, nonlinear, and joint cost functions using shared data, objectives, training budgets, and model-selection procedures.
  2. The observation that a simple linear cost can match or outperform more expressive alternatives, while temporal prediction accuracy does not necessarily correlate with planning performance, is potentially useful for future work in world-model planning.
Critical weaknesses
  1. The paper reports an interesting empirical observation, but provides limited explanation of why linear costs outperform more expressive alternatives or under what conditions this behavior should be expected.
  2. The claim that linear costs should serve as the default baseline for learned planning costs appears stronger than the evidence supports, given the limited set of environments and world models evaluated.
  3. The TwoRoom benchmark is nearly saturated (96–97% success for all learned costs), making it difficult to assess the practical value of additional model expressiveness.
  4. The main conclusions rely heavily on TwoRoom and PushT, particularly PushT. More environments and world-model architectures would be needed to establish generality.
Questions for authors
  1. To what extent do the conclusions hold beyond TwoRoom and PushT?
  2. Have the authors tested additional environments where planning performance is farther from saturation?
  3. See other questions in the weaknesses.
1 comment

Thanks for the authors' rebuttal. The new experiments strengthen the empirical evidence and address several of my concerns.

However, my main concern remains: the paper shows that linear costs often work surprisingly well, but it provides limited insight into why this happens or when to expect it. As a result, I still view the contribution primarily as an empirical observation, so I will keep my original score.

Reviewer_YZm1OpenReview ↗
Rating
3(−1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies how different costs impact the planning performance of the same frozen world-model for a navigation and a manipulation task. They claim that a linear cost learnt to predict elapsed time between a given frame and target frame is a better baseline for planning algorithms to beat than other more complex cost forms. They support this claim by running experiments against a non-linear and joint scorer (from baseline paper) with the same elapsed time objective. They train other cost families by running a grid search across hyperparams to eliminate artifacts from training suboptimalities. The paper also compares learnt linear cost against whitening-based cost and linear cost learnt from shuffled input, goal and label pairs.

The paper advocates for linear mapping as an introspectable way of stretching and rotating the existing axes of a the latent from a learnt world-model and specifically temporal objective as a way to highlight those more relevant to goal progress.

Critical strengths
  • Very good controls against linear cost, e.g. whitening and shuffled objective
  • Adequate complexity of learned costs, i.e. non-linear and joint scorer from an established prior work
  • Position probe as a way to visualize what is happening in higher-dimensional reweighting which helps with conceptual understanding
  • Good experimental practices like hyperparam tuning for other costs
  • Sufficient details for setup and reproduction
Critical weaknesses
  • Unclear if the cost tuning has similar/same budgets. I understand the grid is comparable.
  • TRM has a simpler linear cost than this paper which underperforms heavily. Clearly restating that and differentiating from their own construction is important. Especially as top-level message is opposite from the two papers until one takes a closer read.
Questions for authors
  1. Please address if tuning also uses the same budget.
  2. I expect whitening to fall to raw latent distance behavior for LeWM esp due to SigREG regularization built into its learning. I am surprised that it does the same for latents from PLDM. Any intuition on why?
  3. What is the difference between TRM's geodesic labels and your elapsed time labels which make one objective better than the other? I think it'd be useful to have that comparison here as well, I am not sure if its a full rank linear map which is helping or the specific elapsed time objective. This should soften/alter or bolden the claim.
Reviewer_i87QOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This work studies cost functions for CEM planning learned on top of JEPA latents from a frozen world model. It asks the question: when regressing from latent pairs to absolute time differences, does the mapping being nonlinear help? The answer seems to be that nonlinear maps can result in higher prediction quality, but not better planning, so linear maps are a strong baseline.

More precisely, given the (frozen) latent encodings for the current state zz and the goal gg, the cross-entropy method (CEM) uses the JEPA model to roll out latent variables and scores them according to a cost function, which in the linear case is simply ∣∣A(z^t+H−g)∣∣2/d||A(\hat z_{t+H}-g)||^2/d. CEM iteratively refits the proposal to the best rollouts and executes the final plan. The cost function's parameters (just AA in the linear case) are trained by making it regress to time differences, i.e., ∣∣A(zi−zj)∣∣2/d||A(z_i-z_j)||^2/d should align with ∣ti−tj∣/s|t_i - t_j|/s using a Smooth-L1 loss. This setup was introduced in prior work (TRM), what is new in this paper is questioning whether the cost function being non-linear helps and how much. The paper argues that the baseline that learned planning should beat is a linear cost with an analogous learning, and that non-linear models might be overkill.

Critical strengths
  1. Although not discussed in the paper directly, it corrects a potential over-reading of previous work. The TRM paper might be read to imply: Planning with linear cost function doesn't work, planning with an MLP works, therefore nonlinearity enables planning. This work clarifies that there is something in between TRM’s linear and "diagonal-Mahalanobis" baselines and an MLP: a full linear transformation of the latents (i.e., full-Mahalanobis), and that, under a similar training method, is enough to make planning work. Controls like whitening, shuffled targets, and decoded position show that the advantage is in the temporal supervision (and not just in the use of a linear projection).

  2. It shows that using the temporal loss to select the cost can fail. The linear projection in the PushT experiments, for all configurations, had a higher temporal loss and yet planned better than every nonlinear configuration. That provides caution against what could look like a reasonable criterion.

Critical weaknesses
  1. The empirical results are weak, which makes the observation of the paper mildly interesting but potentially an anecdote.
  • The experiments do not push planning to its limits. E.g., the TwoRoom environment is very simple. In it, all temporally learned costs plan equally well with LeWM (>96%). The experiment is too saturated for the linear vs non-linear comparison.
  • What the findings support is closer to "we haven't found an environment in which a non-linear cost yields more than a marginal benefit". But at the same time, this hasn't been proven nor probed rigorously. There could be a harder environment in which the MLP becomes necessary, while linear works well in simple environments. With the experiments provided, we cannot know if "linear is enough" in general.
  • PushT is only evaluated in one setting (one model checkpoint, one goal offset). It's very difficult to conclude anything general from this.
  • The mechanism behind the PushT inversion (between cost performance and planning performance) is unexplained. (Singh, 2026) shows that the latents can drift off the manifold of real encodings when rolling forward, in which case the cause is the failure of the flexible costs (clearly J, marginally N) at extrapolation. Then it is possible that additional regularization helps. Finding the mechanism and attempting to tackle the problem directly would be more helpful.
  • Only the learned cost is used. Prior work blends it with the raw cost on the latents and finds superior performance. This is left untested.

Overall: experiments on simple problems, without enough diversity or complexity, and the mechanism behind the PushT inversion is left unspecified.

  1. The novelty is very narrow.
  • The whole paper reads like an ablation on TRM's head, the value is in explaining and correcting TRM’s apparent linear failure. The paper does this in Appendix G for its own linear cost, but doesn't connect it to TRM’s baselines.
  • The observation that choosing the cost based on the temporal loss can be suboptimal for planning is related to the observation in (Singh, 2026) that fitting spatial distance better doesn't result in better planning.
  • Therefore, the novelty is limited to observing that, in a narrow setting, a simpler cost than prior works doesn't degrade performance. It doesn't significantly improve performance, it is not tested at scale and diversity in complex scenarios to make sure that the finding is robust, the PushT inversion is not analyzed enough, so it is unclear what the underlying mechanism is.

There are connections with related work: Temporal-Distance-JEPA, Traj-LeWM, Decision-Metric Alignment, Control-Geometry Straightening, Anchored Planning, and TEMPO (particularly the latter two). Some of this work might be too recent to be included experimentally, but it would be worth discussing and connecting with the work presented here, at least in a revision. E.g., TEMPO's nonlinear temporal map raises LeWM TwoRoom from 36% to 99% (75 steps from the goal, cost blended with raw distance). Does the linear map still suffice at that longer horizon?

(Arxiv IDs: 2607.25337, 2608.14125, 2608.18746, 2609.35603, 2609.30036, 2610.04988)

Questions for authors
  • Can you produce results in more complicated navigation cases? Like mazes with loops? How does linear/non-linear hold up in those settings? E.g., TEMPO evaluates frozen LeWM and PLDM on PointMaze medium/large. A nonlinear temporal (hybrid) cost substantially beats raw planning. How well does the linear map hold up there?
  • Is latent drift the cause of the failure of the flexible costs and a linear head is simply extrapolating better? Can you test this? You can e.g., take a held-out episode in which you have logged the latents and δt\delta t and look at the difference in the loss function when you introduce the real latents zz vs the predicted latents, starting with the latent observed before the latest action chunk, and after replaying the latest action chunk using the predictor from the model (imagination run). Is there ranking agreement between executed vs imagination losses?
  • On high-distance pools, TRM shows learned linear failing (15.3%), trained there on geodesic labels. Can you reconcile that with the thesis of this paper?
  • If you use hybrid costs, can you push the MLP performance higher?
  • For J on PushT, what happens if you combine deployment-time symmetrization with the tuned configuration J⋆J^\star? And if you retrain J with symmetric inputs?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.