acceptodds
Under review as a conference paper at ICLR 2027

Back to State: Explicit Physical State World Models for Continuous Control

Abstract

Temporal difference learning for model predictive control (TD-MPC) and its recent variants achieve strong continuous control performance by learning task-oriented latent dynamics and performing short-horizon planning entirely in latent space. For state-based control, however, this removes the physical semantics from imagined trajectories and makes physical state predictions unavailable for deployment-time objectives. Prior work has attempted to recover explicit state predictions from latent world models, but reconstruction-based approaches can suffer substantial performance degradation on challenging high-dimensional continuous control tasks. In this work, we introduce State TD-MPC (S-TD-MPC), which directly propagates the normalized physical state while augmenting it with learned latent features recomputed at each imagined step, providing explicit physical state predictions while retaining strong representation capacity. We further align value learning with MPC through MPC-based Bellman targets and introduce reanalysis via Bellman target bounds to refresh stale planner targets selectively. Beyond the training objective, we study three capabilities enabled by explicit state predictions: zero-shot transfer across tasks with shared dynamics, deployment-time state constraints, and analytic reward evaluation during training. Experiments on challenging continuous control tasks show that S-TD-MPC matches the performance of state-of-the-art methods in the TD-MPC family, achieving an average return 2.2% higher than the strongest baseline, while enabling these constraint-aware and transferable control capabilities.

Primary area:
reinforcement learning
Keywords:
model-based reinforcement learning, world models, continuous control

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_v3SROpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes S-TD-MPC, which directly propagates physical states while recomputing learned latent features at each imagined step. It also introduces MPC-based value targets and selective reanalysis. The main contribution is maintaining competitive control performance while making predicted trajectories available for reward changes and deployment-time constraints. The latent-decoder comparison provides useful evidence for the architectural choice. Although the component ablations and benchmark coverage could be stronger, the results support a practical contribution. I lean toward weak acceptance.

Critical strengths
  1. The paper addresses a clear limitation of latent TD-MPC models: imagined trajectories are difficult to use for objectives defined in physical coordinates. S-TD-MPC retains competitive performance on challenging control tasks while supporting reward changes and deployment-time constraints without model retraining. This flexibility is more consequential than the modest improvement in average benchmark return.
  2. The latent-decoder comparison provides meaningful support for direct state propagation. The baseline retains the same planner-based learning and reanalysis framework, but performs substantially worse on the more dynamic Dog tasks. The downstream comparisons further demonstrate that the representation choice matters beyond standard training returns.
Critical weaknesses
  1. The ablations do not fully separate the architectural and training contributions. The latent-decoder comparison is useful, but the paper does not establish whether auxiliary latent features improve over a comparable state-only model. It also lacks an explicit-state variant trained with the original TD-MPC2 objectives. Consequently, the respective contributions of state propagation, feature augmentation, and planner-based training remain unclear.
  2. The evidence for prediction quality and generality is limited. The evaluation covers seven tasks from two dynamical systems, without directly reporting one-step or multi-step state prediction errors. Such measurements would be particularly useful because the downstream experiments extend the planning horizon substantially beyond the training horizon. Constraint handling also relies on penalties and exhibits a return–constraint trade-off, so the results should not be interpreted as establishing reliable constraint satisfaction.
Questions for authors
  1. Do you have results for a state-only variant without auxiliary latent features, using comparable network capacity and the same training procedure? This would clarify the contribution of the combined representation.
  2. How does the explicit-state architecture perform with the original TD-MPC2 value and policy objectives? This comparison would help distinguish architectural gains from changes to training.
  3. Can you provide multi-step physical-state prediction errors for S-TD-MPC and the latent-decoder baseline, particularly over the longer horizons used in downstream evaluations? This would help explain the observed differences and assess their practical limits.
Reviewer_YRYwOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes State TD-MPC (S-TD-MPC) for state-based continuous control. The core idea is straightforward: predict physical states directly and recompute auxiliary latent features at each imagined step. The authors also introduce MPC-based value targets and selectively refresh cached planning targets using their age and the critic's action sensitivity. On seven state-based DMC tasks, the reported returns are strong but somehow broadly to baselines such as MRS.Q.

Critical strengths
  1. The method and the main claims are clearly presented. I found the paper easy to follow, and the authors explain the design choices well.

  2. The experimental evaluation is fairly thorough, with several strong baselines and relevant ablations. The method does achieve strong control performance across the evaluated tasks.

  3. I particularly liked the analysis in Section 5.4 and Appendix C. The transfer and constraint experiments make the usefulness of explicit state predictions concrete. The asymmetric transfer results and the different effects of leg and neck constraints also provide some interesting insight into when this flexibility works well and what it costs.

Critical weaknesses
  1. My main concern is what is actually new about the modeling design. The inputs are already physical state variables, so the need for a separate auxiliary latent is not obvious to me. Although the model directly propagates states (as the title, back-to-state), the paper does not establish what the learned latent add over a straightforward state-space dynamics model which is normal previously. At present, this feels more like revisiting an existing approach than establishing a new modeling contribution. The latent-decoder baseline does not resolve this concern. A controlled comparison against direct state prediction without the auxiliary feature branch, with comparable capacity and the same training framework, or a convincing theoretical justification, would help establish why this design is needed.

  2. The performance gains over strong baselines also look fairly limited. The method certainly performs well, but I do not find these gains particularly compelling especially compared with the MRS.Q. These results support competitiveness, while providing limited evidence of a meaningful performance advantage from the proposed design.

Questions for authors

See Weaknesses.

Reviewer_5bV3OpenReview ↗
Rating
3
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

TD-MPC methods (TD-MPC, TD-MPC2 and their variants such as BMPC and BOOM) learn a task-oriented latent dynamics model and plan with short-horizon MPC on top of a TD-learned value. They are among the strongest agents for high-dimensional continuous control, such as the DMC Dog and Humanoid tasks. Their weakness is that the imagined rollouts live in a learned latent space with no physical meaning, even when the input is already a physical state vector. The planner therefore cannot check a state constraint, plug in a known reward, or reuse the dynamics for a new task, and adding a decoder to recover the state is reported to hurt performance on hard tasks (Section 1). This is the motivation for the paper.

Why a new dynamics model: The paper asks whether a TD-MPC-family agent can keep its rollouts in physical state and still match the latent methods. It proposes S-TD-MPC, whose world model predicts the next normalized physical state directly, so that predicted states can be used for constraints, analytic rewards and task transfer at deployment (Section 4).

Modeling: The dynamics take the concatenation of the normalized state and a latent feature computed from it, z = h(s̃), and predict a bounded residual on the normalized state. At every imagined step, z is recomputed from the predicted state and fed, together with the state, to the dynamics, reward, value and policy heads; the paper credits this feature with additional representation capacity. Direct state supervision over a 3-step unroll replaces TD-MPC2's latent consistency loss; the two-hot reward and value losses are kept, and the policy prior is trained to match the MPPI search distribution, following BMPC (Sections 4.1-4.2, Equations 2-4).

Value learning: The critic is trained on the actions the planner actually takes, rather than the policy's (Equations 6-7). Stale planner targets are refreshed in order of how much they are likely to bias the value (Equations 8-9, Theorem 2).

Experiments: On the DMC Humanoid and Dog tasks, S-TD-MPC matches or slightly exceeds the strongest latent TD-MPC methods, including BOOM, MRS.Q, BMPC and TD-MPC2 (Table 1). Because its rollouts stay in physical state, the same frozen model transfers to new tasks with analytic rewards better than a latent-decoder variant, and it can plan under state constraints added at deployment (Section 5.4).

Critical strengths
  • Paper shows learning dynamics over physical states itself can help in achieving performance similar to TD-MPC while being efficient.
  • The physical state model is easy to understand and adopt for any change in objective.
Critical weaknesses

1. It is unclear whether the latent feature z helps the dynamics predictor.

The dynamics take the normalized state together with a latent feature recomputed from it, z = h(s̃), and the paper credits z with "additional representation capacity" [p.4, Section 4.1; p.7, Section 5.2], but no experiment removes z from the dynamics predictor. Since z is computed from the state itself, it may add little to state prediction and matter mainly for the reward, value and policy heads, where a task-specific feature is useful. An ablation that removes z from the dynamics only, keeping it in the other heads, and reports multi-step state-prediction error alongside return on a few Dog and Humanoid tasks, would show whether the dynamics get any better because of z.

Questions for authors

I would like to see an expansion of the experiments to the full dm_control suite. I also have two questions:

  1. [Clarification] How does S-TD-MPC differ from state-space model-based methods such as PETS and MBPO?
    • Section 2 places it "between these two designs" because it retains learned latent features [p.2, Section 2]. If z does not help the dynamics (Weakness 1), what remains is TD-MPC's value learning and planning on a state-space model; please state that difference, and ideally compare against a PETS- or MBPO-style model on a few Dog and Humanoid tasks.
  2. [Clarification] Does the latent feature z improve the dynamics predictor, or only the reward, value and policy heads?
    • Since z = h(s̃) is computed from the state itself [p.4, Section 4.1], please report S-TD-MPC with z removed from the dynamics only, giving multi-step state-prediction error and return on a few Dog and Humanoid tasks (Weakness 1). If prediction error and return are unchanged, z's role is task-specific features for the heads rather than better dynamics.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.