acceptodds
Withdrawn from ICLR 2027

Prediction Is Not Control: Equivariance Enables Planning in Weight-Space World Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

A world model lets an agent plan by imagining how its environment responds to actions. For reward-free, goal-conditioned planning, such models arrive by two routes: large models pretrained on web-scale data, which control well but are opaque, and from-scratch functional models whose state is the weights of an implicit neural representation, inspectable and pretraining-free but not yet shown to plan. We show the second route fails not on prediction but on controllability; over the same INR-weight state, non-equivariant and latent-code operators match an equivariant operator's rollout PSNR yet plan no better than doing nothing. We trace this to symmetry where only a permutation-equivariant operator learns an action map that survives the weight-space gauge and can be inverted by a planner. We distill this into an offline diagnostic, the action-response, that predicts controllability at r=0.91. Composed with a classical planner on occupancy read off its state, the equivariant model is competitive-to-ahead of a pretrained state-of-the-art world model in distribution and more than 2× ahead out of distribution, from scratch and without reward. Controllability, not prediction, is what makes a world model useful for planning.

Primary area:
applications to robotics, autonomy, planning
Keywords:
world models, planning, controllability, implicit neural representations, neural functionals, equivariance

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_FooQOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

INFun-WM represents observations with fitted SIREN weights and predicts their changes after an action using a permutation-equivariant NFT. It plans with CEM and, for navigation, A* waypoints. Models with similar image prediction quality show different planning success. The authors attribute this gap to equivariance and propose an action-response diagnostic.

Critical strengths
  1. The gap between prediction and control is a useful observation. At C=2, INFun-WM and FLAT achieve similar rollout PSNR, 15.21 and 15.11 dB, but success rates of 65% and 10%. The analysis of movement direction helps explain why similar image scores can hide errors that matter for planning.
Critical weaknesses
  1. The experiments do not isolate equivariance as the cause of better control. Architecture and state representation also change. The comparison should match capacity, training and neuron alignment. Table 4 tests canonicalization for static safe-action prediction, but not dynamics and planning.
    The NOVA-style MLP has -107.7/-113.7 dB PSNR (Table 3), so it cannot support the matched-prediction claim. It uses this paper's per-trajectory INR fitting, whereas NOVA uses a shared encoder and base weights. Its failure therefore does not establish that original NOVA is unsuitable for control. Also, networks representing the same image need not differ only by neuron order. Equivariance ensures consistency under reordering, but even a zero-update operator is equivariant. It does not guarantee correct action effects.

  2. The contribution relies on the control experiments, whose interpretation remains uncertain. NFN/NFT already learn equivariant INR edits [1,2]; NFT Section 5.2 predicts residual SIREN edits using image losses. NOVA evolves INR weights using inferred latent actions [3], and prediction/control mismatch is established in model-based RL [4]. The distinct step here is to condition an equivariant weight operator on environment actions and test closed-loop planning.
    The action-response score is high for LATENT (0.77/0.79) despite poor control, and Figure 4 excludes clean PointMaze from the correlation fit. Figure 10 describes different inputs for the DINO-WM comparison: clean egocentric observations for INFun-WM and native allocentric RGB for DINO-WM. Sharing A* does not remove this difference. Success also falls to 23% with four objects, with failures under texture and contact. These limits leave practical value uncertain, especially given the cost of fitting each observation.

References:
[1] Permutation Equivariant Neural Functionals. NeurIPS 2023. https://arxiv.org/abs/2302.14040
[2] Neural Functional Transformers. NeurIPS 2023. https://arxiv.org/abs/2305.13546
[3] Render, Don't Decode: Weight-Space World Models with Latent Structural Disentanglement. 2026. https://arxiv.org/abs/2605.06298
[4] Objective Mismatch in Model-based Reinforcement Learning. L4DC 2020. https://arxiv.org/abs/2002.04523

Questions for authors
  1. With matched capacity and training, does the control advantage remain over alternatives using aligned neurons or shared initialization? Please test dynamics and planning, beyond the static readout in Table 4.
  2. Do the gains over DINO-WM remain with matched observations, actions, goals and planning budgets? Figure 10 indicates that the observations differ.
  3. Why does action-response fail for LATENT, and why is clean PointMaze excluded from the correlation fit? Does the denominator use true or predicted transitions? Does this diagnostic predict success on held-out environments?
Reviewer_XdNUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The authors study reward-free goal-conditioned planning settings where the tasks is to reach an arbitrary target observation. They differentiate the field of world model planning by 1) leveraging large amount of data to pretrain latent models such as DINO-WM which can plan well but are opaque or from-scratch functional models that are interpretable and pretraining-free but have not yet to be shown to plan. The authors argue that these from-scratch function models fail at controllability rather than prediction and the limiting factor is symmetry of the transition operator.

The authors propose INFun-WM, an INR-weight world model whose transition is a permutation-equivariant neural functional operator where each frame is fit with SIREN and a permutation-equivariant neural operator rolls the weight sunder actions and plans using CEM-MPC. The paper reports that while the baselines like FLAT, NOVA-style MLP, DINo match INFun-WM's rollout PSNR, they fail to plan, supporting the argument that controllability is different from prediction and is what is necessary for planning. The also propose an action-response diagnostic which acts as an inexpensive, offline predictor of a model's controllability that correlates highly with the real planning success.

Critical strengths

The paper offers a clean experimental design and useful probing. Fixing the INR-weight state while varying the operator design highlights the importance of the choice of operator for determining whether a model can plan and not the impact of the state representations. The directional probe is useful that exposes failure modes that the PSNR metric hides. Additionally, this paper isolates controllability as the evaluation criteria separate from prediction capabilities in action-conditioned planning in weight space world models which prior work has not done.

Critical weaknesses

W1. The central claims seem a bit overstated and confounded. PSNR measures error over pixels on a log scale and since these are 64x64 frames with mostly black with a few small white sprites, the PSNR is dominated by the background and could barely penalize a sprite in the wrong place. Differences in PSNR are therefore limited and not very informative of prediction quality here. For example, if you take the results in figure 3 that show an agent's induced displacement from the rendered prediction versus the true environment, latent and flat have extremely poor scores which could indicate that these operators put the sprites in the wrong place yet their PSNR scores are similar. The claim that the model can predict well and still be uncontrollable might not be the case if the evidence for prediction quality is just PSNR. A different prediction metric or different visual benchmarks may be needed to truly support the claim that FLAT and LATENT predict as well as INFun-WM yet have low controllability.

Additionally, as the paper acknowledges LATENT scores 0.77 in the action-response metric yet does not work for planning. Because the action-response measures only the magnitude of the action's effect, it may not be a good offline predictor of a model's controllability as claimed. The directional probe in figure 3 requires the the true environment's response to each probe action, limiting its offline prediction capabilities.

Finally, the authors claim competitive-to-ahead performance against DINO-WM, a SOTA pretrained world model. However, INFun-WM fails on PointMaze's textured mujoco render and is instead evaluated on clean-render PointMaze viewed egocentrically (Figure 9) while DINO-WM uses the native allocentric RGB observation (Figure 10). Although it is true that the A* scaffold and absence of proprioception are matches, the observations differ in both texture and viewpoint. It seems unfair to claim competitiveness to DINO-WM as a contribution when the setup is different. INFun-WM is clearly given observation setup where it works where DINO-WM is given a setup that INFun-WM has already shown to fail on.

To summarize, the central claim seems to be overstated on different accounts but feel free to explain/convince otherwise.

Questions for authors

Q1. The non-floor points in Figure 4 appear to match Table 2, which corresponds to INFun-WM. Is the the r=0.91 score for action-response diagnostic computed for INFun-WM alone across environments? If so, doesn't this indicate which environments INFun-WM works well for and not which models are controllable? Could the authors add FLAT, MLP, and LATENT to Figure 4 and report the correlations across operators and report the action-response values for textured PointMaze and PushT to show if the action-response diagnostic holds for all models?

Q2. How are the actions in the train and test trajectories generated and are the probe actions in figure 3 drawn from the same distribution? Are the probe actions similar to the actions the model is trained on or different? For example, does the data come from random actions over the same grid or could it come from a policy or larger grid? To test whether FLAT and LATENT genuinely predict as well as INFun-WM could the authors evaluate on held-out transitions and report the error between predicted and true sprite positions or PSNR computed over only the sprite pixels. As a reference point, it would also help to report the PSNR of just a black empty frame as a floor baseline PSNR number.

Reviewer_DfxeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper is framed in the context of goal-conditioned planning from pixels, but rather than representing the state in pixel space or in a learned latent space, they work over weights of a small network (an idea that stems from the SIREN framework). This network is fitted to each frame and is meant to represent the underlying states through the chosen weights of that network. Then, the same way you would in any other model-based planning setup, you then learn a transition operator. Again, rather than mapping raw observation space (raw observations to raw observations or state to state), it actually maps a given set of weights of a fitted network to another set of weights. Now, the headline of the paper is a fact that has been studied extensively in the reinforcement learning literature and beyond, even in system identification. In system identification, it's what underlies control-oriented system identification. It has also been an idea widely studied under the umbrella of decision-focused learning, or decision-aware learning, or value-aware learning in the reinforcement learning literature. The fact that a model of high accuracy need not be a good model for planning is definitely not a new fact to the community. That being said, the particular framing of the paper and the uniqueness of it, as far as I know, is that they study that question in a setting where the representation of the space of the state is through the weights of the fitted network: an implicit representation . And then, when asking the question "What makes a good model in that setting?", the authors then argue a particular phenomenon: the model response will match how the scene actually moves only if the transition operator respects a permutation of the hidden neuron without changing the corresponding image. Making the transition operator equivariant in this manner is shown to provide better prediction capabilities compared to other models, which would not implement that kind of inductive bias.

Critical strengths

Learning decision-focused models, or models that are good for control, is still very much an open question. Many of the efforts over the last few years have been based on the idea that there might be a better loss function that favors such models, but very few of these methods turned out to be actually used in practice, as far as I know. The paper asks an important question and might be providing a new angle or mechanism on this important question.

Critical weaknesses

Now, the control side of the argument is really more framing rather than actually a method. The title might suggest otherwise, but really, it's more of a framing. What I mean here is that this paper is not really providing a method. They really train every model for prediction and, in their experiments, then look for controllability. So there's no explicit control-aware objective, and the planner is not really taken into account in the training loop the way it would be done in a control-oriented design. Now, the paper does cite the value coherence and objective mismatch in the background, which is good, but there is substantial literature beyond that. But a claim, such as one made by the authors that controllability is a criterion that is rarely studied and isolated on its own, is not quite true because there is an extensive literature on that problem. While you would expect the weight space part of the paper and the controller-oriented part to go hand in hand together, they're more like two separate ideas that ended up in the same paper. I think it would have been cool to see a paper that says, "If you want to do control-oriented or decision-focused learning, weight-space models or implicit representation is the way to go." But this is not quite what's happening here.

Also, in terms of terminology, there might be a bit of a mismatch, or perhaps a source of confusion, with the system identification and control literature. When we're talking about controllability, really, the alters mean how strongly and in which direction the predicted image is going to go under the action. This is much closer in spirit to the concept of a gain rather than the concept of controllability, at least in Kalman's sense.

I would have liked to see evidence that space models are the right class of models, and this could have been done in the baselines. The choice of baselines: the comparisons are currently mostly amongst models in that same class. The issue with permutation really stems from the protocol because, at each trajectory, a network is going to be fitted from a random initialization. But if you were to fit from the same shared initialization, or if you were to sort the neurons in some chosen order, that could also act as a good mechanism for the problem that is raised in the paper. This is not tested as a simple baseline for control. In that same vein, it would have been good to see a pixel-space model along with the weight-space models.

Finally, I think it would be important to substantiate the references a bit. In particular, when saying that learning a dynamics model and acting over it is a long-standing recipe, the authors choose to cite Ha and Schmidhuber (2018). This is actually a very recent paper, and work on model-based reinforcement learning goes way beyond that and goes even way beyond model-based RL in the system identification and control literature. You would want to at least cite also Sutton's Dyna from 1990 or the work by Schmidhuber on recurrent world models in the '90s. In the same way, Ha and Schmidhuber (2018) also build on Jordan and Rumelhart's model from '92. There is also PILCO in 2011.

Questions for authors

So when making a point that prediction loss is the wrong criterion for control, I think you have no choice but to compare against control-aware objectives and methods in the literature on decision-aware, decision-focused, value-aware learning. You'd want to at least compare it with your model. I'm not saying that these methods need to be better than yours, and in fact, they're quite tricky to get working. From experience, your method might work even better, but it needs to be shown.

So what would happen if you were to add a non-equivariant transition on the weights that you fit from a shared initialization, or what would happen if you were to put them in some chosen order? What would happen if, for the same capacity as your weight space model, you worked with a pixel space model instead?

Reviewer_eBsYOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

Existing world models may predict future observations accurately but fail to support effective control due to inaccurate action-conditioned dynamics. To address this problem, the authors propose INFun-WM, which employs a permutation-equivariant operator to learn dynamics in the INR weight space. Although trained solely with an image prediction loss, INFun-WM demonstrates more accurate action responses and better performance on downstream control and planning tasks

Critical strengths

The paper is well-motivated. World models need to capture the relationship between actions and future states to facilitate downstream planning and control. Investigating the discrepancy between prediction accuracy and controllability is a meaningful research direction.

Critical weaknesses
  1. The presentation is poor. The writing is repetitive and unnecessarily complicated, making the paper difficult to read and understand. Some figures are also not sufficiently clear or informative. The paper would benefit from a more concise and better-organized presentation.
  2. The experimental evaluation is limited. Most successful experiments focus on simple 2D environments with a small number of objects, without convincing results on more challenging scenarios. The method also fails on tasks involving complex observations or contact-based dynamics. The current experiments resemble extended toy demonstrations, which may illustrate the proposed mechanism but are insufficient to support the broader claims of the paper.
  3. The necessity of the proposed method is not sufficiently justified. The proposed approach involves learning dynamics in a high-dimensional INR weight space, which introduces additional complexity. However, the paper lacks comparisons with simpler alternatives, such as directly learning action-conditioned dynamics in an explicit state space. It is unclear whether the proposed weight-space formulation is necessary for the evaluated tasks.
Questions for authors
  1. Can the proposed method be extended to more challenging environments with complex observations and dynamics? What are the main obstacles to applying it to realistic control tasks?
  2. Why is it necessary to learn dynamics in the INR weight space? Could simpler action-conditioned dynamics models achieve comparable performance on the evaluated tasks?

Comments on the paper

1 comment

Thank you for all the reviewers' time and feedback. We are withdrawing this submission.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.