acceptodds
Under review as a conference paper at ICLR 2027

Slot-MPC: Goal-Conditioned Model Predictive Control with Object-Centric Representations

Abstract

Predictive world models enable agents to model scene dynamics and reason about the consequences of their actions. Inspired by human perception, object-centric world models capture scene dynamics using object-level representations, which can be used for downstream applications such as action planning. However, most object-centric world models and reinforcement learning (RL) approaches learn reactive policies that are fixed at inference time, limiting generalization to novel situations. We propose Slot-MPC, an object-centric world modeling framework that enables planning through Model Predictive Control (MPC). Slot-MPC leverages vision encoders to learn slot-based representations, which encode individual objects in the scene, and uses these structured representations to learn an action-conditioned object-centric dynamics model. At inference time, the learned dynamics model enables action planning via MPC, allowing agents to adapt to previously unseen situations. Since the learned world model is differentiable, we can use gradient-based MPC to directly optimize actions, which is computationally more efficient than relying on gradient-free, sampling-based MPC methods. Experiments on simulated robotic manipulation tasks show that Slot-MPC improves both task performance and planning efficiency compared to non-object-centric world model baselines. In the considered offline setting with limited state-action coverage, we find that gradient-based MPC performs better than gradient-free, sampling-based MPC. Our results demonstrate that explicitly structured, object-centric representations provide a strong inductive bias for controllable and generalizable decision-making. Code and additional material will be released at https://slot-mpc.github.io.

Primary area:
applications to robotics, autonomy, planning
Keywords:
model predictive control, world models, object-centric representations

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_pgGPOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents Slot-MPC, a framework for visual goal-conditioned control combining a SAVi-based object-centric encoder, an action-conditioned latent dynamics model, and gradient-based model predictive control.

Critical strengths
  • A coherent approach to planning with compact object representations. The method integrates perception, action-conditioned prediction, and visual goal matching in a straightforward framework.
  • Clear benefits in some settings. MPC improves Button Press success from 0.60 to 1.00 over the policy-only variant. The visual-shift experiments are also encouraging, especially because the goal images retain their original appearance.
Critical weaknesses

The methodological novelty appears limited. Slot-based world modeling is well established in prior work such as SlotFormer and PlaySlot, while policy-initialized gradient-based MPC has already been explored in Dream-MPC. The contribution therefore appears primarily to be their integration for offline, visual goal-conditioned control. Although this combination could be valuable, the paper does not sufficiently establish the additional technical insight or empirical benefit beyond these existing ingredients, particularly given the different planning horizons and search configurations used in the optimizer comparison.

Questions for authors

What is the main advance beyond combining existing slot-based world models with policy-initialized gradient-based MPC, and which controlled experiment best demonstrates this advance?

Reviewer_NL3MOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes Slot-MPC, which combines object-centric representations, an action-conditioned world model, and gradient-based MPC for goal-conditioned planning from visual observations. Experimental results demonstrate strong task success rates and planning efficiency across several robotic manipulation tasks.

Critical strengths
  1. The paper addresses a meaningful research problem with a clear overall approach, integrating object-centric representations into world model-based visual planning with gradient-based MPC.

  2. The method achieves strong experimental results, outperforming several baselines across multiple tasks, particularly in planning efficiency. The paper also provides ablation studies and analyses of latent-space distances.

Critical weaknesses
  1. Limited novelty and insufficient analysis of the underlying mechanism. Slot-MPC primarily combines existing components, including object-centric representations, an action-conditioned world model, behavior cloning, and gradient-based MPC. Although Figure 2 shows that slot-based latent distances provide a smoother measure of progress toward the goal, this does not directly demonstrate that they are more suitable for gradient-based optimization. Further analysis is needed to explain why object-centric representations improve the effectiveness of gradient-based MPC.

  2. Potential fairness concerns in the MPC comparison. According to Appendix C.4, gradient-based MPC uses planning horizons of 1, 2, or 8, whereas MPPI uses a horizon of 15. Different horizons may affect prediction errors and optimization difficulty, making it difficult to isolate the contribution of the optimization method itself. Comparisons under matched horizons would help clarify the results.

  3. Unclear contribution of MPC beyond the BC policy. Slot-MPC uses a BC policy that is not conditioned on goal slots to initialize action sequences. Table 2 shows that removing MPC still achieves relatively high success rates on tasks such as Stack, whereas removing policy initialization causes a substantial performance drop. This raises questions about how much of the performance comes from the BC policy versus goal-conditioned planning. Furthermore, the robustness experiments mainly focus on visual distribution shifts, with limited direct evaluation of adaptation to different goal configurations.

Overall, the paper presents a clear approach and strong experimental results, particularly in planning efficiency. However, the underlying mechanism behind the performance improvements requires further analysis, and the different planning horizons raise concerns about the MPC comparison. Additionally, the contribution of goal-conditioned MPC beyond the BC policy, especially in adapting to different goals, needs to be more clearly demonstrated.

Questions for authors
  1. Could the authors provide further analysis or experiments to explain why object-centric representations are particularly beneficial for gradient-based MPC?

  2. What motivates the different planning horizons used for gradient-based MPC and MPPI? How do the methods compare under matched horizons?

  3. Could the authors further clarify the contribution of MPC beyond the BC policy? For example, evaluating different goal configurations within the same environment, with and without MPC, could help demonstrate the additional goal-adaptation capability provided by planning.

Reviewer_ajDoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Slot-MPC does goal-image-conditioned manipulation by planning in a slot space, trained offline and without rewards. SAVi splits each image into 4 or 5 slots, a transformer world model predicts future slots given actions, and a behavior-cloning policy, which does not see the goal, proposes an initial action sequence. The planner then takes 1 to 3 gradient steps on the Hungarian-matched distance between predicted and goal slots, executes the first action and replans. On five Meta-World and robosuite tasks it has the highest success among the compared methods, for example 1.00 on Button Press, 0.78 on Stack and 0.26 on Square, and one planning step takes 0.02–0.07 s, against about 144–145 s for DINO-WM.

Critical strengths
  1. Planning is fast. Table 3 reports 0.02–0.07 s per step, against about 144–145 s for DINO-WM and 0.89 s for LeWM, and Slot-MPC has the best success on all five tasks, although on four of them its Wilson interval in Table 1 overlaps with the next method's. Planning cost is a real obstacle to using world models for control, and this is the paper's clearest practical result.
Critical weaknesses
  1. Most of the performance comes from the behavior-cloning policy, and the policy does not see the goal. Table 2 shows that without MPC, success drops by only 0.07, 0.07, 0.03 and 0.02 on four out of five tasks except for Button Press gains a lot,from 0.60 to 1.00. On Meta-World, Table 5 sets the planning horizon to 1 with a single gradient step. So in every task the goal image affects the executed actions only through the planner's gradient steps, one per control step or three on Square. The paper never tests whether the method can reach different goals from the same scene; the sub-goals in App. E.2 are taken from expert trajectories. As it stands, the experiments do not support the framing as goal-conditioned, long-horizon planning.

  2. The comparisons behind the paper's two main claims, that slots beat holistic representations and that gradient MPC beats sampling-based MPC, are confounded. MPPI is run with a horizon of 15 and one configuration for all tasks, while gradient MPC uses horizons of 1 to 8 set per task, as Tables 5 and 6 show. DINO-WM and LeWM run with default hyperparameters and no policy prior, and the paper says larger CEM budgets did not help them but reports no numbers for this. EC-Diffuser and LPWM, in contrast, were tuned. Input resolutions also differ between 64, 128 and 224 pixels. The only holistic-encoder ablation uses a simple convolutional autoencoder. Before the gain can be credited to object-centric representations, I would need to see a pretrained holistic encoder such as DINOv2 with the same policy prior and gradient MPC.

Questions for authors

Questions that could change my score:

  1. Can Slot-MPC reach different goals from the same initial scene, for example a different stacking order or target position? This is the main thing I would need to see to raise my score.
  2. How were the per-task horizon, number of iterations and step size in Table 5 chosen, and were evaluation episodes used for this? How many training seeds were run?
  3. Can you compare gradient MPC and MPPI at the same horizon, and add DINOv2 features with the same policy prior and gradient MPC?
  4. Some numbers disagree. In Table 4, the w/o-MPC Light Direction entries, 0.42 and −23%, do not match the w/o-MPC Button Press value of 0.60 in Table 2, since 0.42 against 0.60 is a 30% drop. Two columns of Table 4 also contradict the claim in Sec. 4.5 that the ablated variants degrade more: under Object Color, w/o object-centric drops 9% against 17% for Slot-MPC, and under Camera View, w/o MPC drops 0% against 7%. Sec. 4.4 mentions an IoU cost that is not in Table 8.

Not affecting my score: the slot distance drops sharply in the first frames in four of the five panels of Figs. 2 and 10, and it does not track progress over most of the Square trajectory or on Stack Three, where it is no better than the holistic distances, so the claim in Sec. 4.2 should be narrowed.

Comments on the paper

1 comment

Dear Reviewers,

I would like to kindly remind everyone to review the authors' responses and provide your feedback.

If you have any further questions or concerns regarding the authors' responses, please feel free to raise them early, so that the authors have sufficient time to address them before the rebuttal period ends.

Thank you all very much for your time and effort! And, have a nice weekend. :)

Best regards,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.