acceptodds
Under review as a conference paper at ICLR 2027

Diagnosing and Repairing World-Model Errors for Planning

Abstract

World models guide planning by predicting action consequences, yet lower average prediction error need not improve action selection. Relative errors in predicted returns can reverse candidate rankings even when average prediction error is small. We develop a support gate that couples penalties on positive predicted rewards to state and action support. A complementary pairwise action-effect loss supplements prediction training by matching differences between real next-state outcomes of alternative actions from the same state. Our analysis connects selection loss to relative score errors and explains how the two designs modify planning scores and relative outcome predictions. The gate reduces CartPole mean normalized regret from 0.676 to 0.077 without retraining, and improves return in 20 of 27 models across nine public-data settings. Component comparisons show further gains from pairwise training with the gate on DoorKey and CartPole, with the full combination performing best among the evaluated configurations. An independent acquisition study under matched interaction budgets finds that additional data can improve control beyond continued training.

Primary area:
learning on time series and dynamical systems
Keywords:
world models, model-based reinforcement learning, model predictive control, support-aware planning, pairwise supervision

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_AqwaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the mismatch between world-model prediction accuracy and planning performance. It proposes two complementary methods: a support gate that penalizes positive predicted rewards based on state-action data support, and a pairwise action-effect loss that supervises differences between action outcomes from the same state. Experiments on DoorKey, CartPole, and MuJoCo environments demonstrate improvements in planning performance, particularly when combining the support gate with pairwise training.

Critical strengths
  1. The paper addresses an important issue in world-model planning: low prediction error does not necessarily lead to better action selection. Its analysis of relative prediction errors provides useful motivation for decision-oriented model repair.
  2. The proposed support gate improves planning performance without retraining the world model. Experiments across multiple environments and component comparisons further demonstrate the potential benefits of combining support-gated planning with pairwise supervision.
Critical weaknesses

Recent studies including CoCo [1], CQM [2], and AD-WM [3] have explicitly addressed action-dependent prediction and relative action effects in world models. In particular, CQM directly learns differences between counterfactual action outcomes, closely related to the proposed pairwise loss. The paper does not sufficiently distinguish its contributions from these approaches.

[1] Overcoming Statistical Bias in Action-Controllable World Models, arXiv:2608.04653.

[2] Counterfactual Quotient Models: Learning What Actions Change, Not What the World Does, arXiv:2608.22092.

[3] AD-WM: Action-Discriminative World Models for Counterfactual Model Predictive Control, arXiv:2609.30264.

Questions for authors
  1. What are the key methodological differences between the proposed pairwise action-effect loss and existing approaches such as CoCo, CQM, and AD-WM? Could the authors clarify what limitations of these methods the proposed loss addresses and provide comparisons where feasible?
  2. Could the authors provide more controlled ablations to isolate the contributions of the proposed mechanisms? Specifically, how does the support gate compare with additive penalties using identical state-action support information, and does pairwise supervision provide consistent improvements under matched training-update budgets?
Reviewer_Qj9aOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines how world-model errors affect the actions chosen by a planner. It introduces a support-based gate on positive predicted rewards and a training loss on differences between same-state action outcomes. The analysis explains their motivation under fixed-candidate and linear-continuation assumptions. Experiments show substantial CartPole gains and improvements in seven of nine Minari settings. A separate acquisition study finds that new data can improve HalfCheetah control beyond continued training, while CartPole results and the extra benefit of distributing branches along trajectories are less clear.

Critical strengths
  1. The focus on relative candidate errors is useful. The paper explains why average prediction error may miss errors that change decisions, and Appendix A states the limits of the analysis carefully.
  2. The fixed-model experiments make it possible to study changes in planning without also changing the trained model. The CartPole regret reduction from 0.676 to 0.077 is substantial. The Minari results also show individual-model outcomes and failures, which helps assess consistency (Tables 1–3 and 7).
  3. The acquisition study has several valuable controls. It counts navigation within the 1,020-transition budget, compares against further training on old data, and uses new dataset/model sources for confirmation. The paired intervals in Appendix D support a measured interpretation of the results.
  4. I appreciate the explicit discussion of limitations. The paper distinguishes state errors from score errors and does not present its local analysis as a guarantee for complete closed-loop behavior.
Critical weaknesses
  1. The key comparisons change more than the proposed mechanism. Pairwise training adds Adam steps, so its improvement in Table 4 could partly reflect additional optimization. The gate comparison also adds state-support information that the additive baseline does not use (Appendix B.2). The paper acknowledges both points, but they still prevent a clear attribution of the gains. Please add extra ordinary-loss updates using the same labels and an additive baseline using the same joint support score, with a comparable calibration budget.
  2. The component results need a verified experimental record. Table 6 states that its settings still need to be checked against the updated Table 4 runs, and the seed was set after model construction (p. 15). Unfortunately, this makes it difficult to reproduce a central result in the paper. Table 4 also gives no paired uncertainty for the full recipe's improvement over gate-only, from 0.077 to 0.045 in CartPole regret. Verified run configurations, initialization records, and paired results would make the evidence much easier to assess. Results across independent datasets would clarify its scope beyond the shared original dataset.
  3. It is unclear how often the support gate behaves as intended in the actual runs. Appendix A.2 explains that continuous-density scores can be negative, that a negative calibration mean reverses their normalized ordering, and that the unclipped gate can amplify positive rewards or turn them negative. I am not assuming that these cases invalidate the reported experiments. However, the observed calibration means and rollout score distributions are needed to judge their relevance. If these cases occur frequently, a nonnegative or clipped alternative would be a useful direct check.
  4. The Minari improvements start from very poor absolute performance. Normalized regret remains close to one in every setting after gating, and Hopper-medium return falls from 67.98 to 5.29 (Tables 2–3). This does not erase the gains on fixed checkpoints, but it limits what they establish about practical repair. Please explain the weak base performance and test whether gating still helps at least one competent fixed checkpoint. A behavior-policy reference would provide useful context without requiring a large new benchmark study.
Questions for authors
  1. Does pairwise supervision still help when ordinary training receives the same number of updates? Does the gate still beat an additive penalty using identical joint support and comparable calibration?
  2. Can you provide verified settings for Table 4 and the paired effects of Full versus gate-only and Full versus Planner + gate? How stable are these differences across independent datasets?
  3. What calibration means and support-score ranges were observed? How often were positive rewards amplified or made negative, and would a nonnegative or clipped gate change the outcome?
  4. Why are the Minari regrets so high, and does the gate improve a competent checkpoint? A behavior-policy comparison and an explanation of the Hopper-medium regression would help interpret these results.
Reviewer_3LnwOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper argues that reducing world-model prediction error does not always lead to better control performance, thereby addressing the known objective-mismatch problem described in [1]. The authors propose two fixes. The first is a “support gate” that penalizes positive predicted rewards at planning time in a CEM-based MPC, using a state-distance term plus a local action-density term calculated from the training data, which they call the support score. The second is a pairwise loss that fits differences between the next states of two actions taken from the same simulator snapshot.

Experiments use 5-member MLP ensembles with CEM planning applied to HalfCheetah, Hopper and Walker2d from the offline RL benchmark Minari, as well as to CartPole Swingup from the DeepMind Control Suite (DMC). Breadth-first search planning is used for discrete-action experiments in DoorKey. A third part addresses whether one can “repair” a model with more data collection.

Critical strengths

I am afraid I could not identify a critical strength that would support acceptance.

Critical weaknesses
  1. Weak experimental results. The Minari results are close to the level of a random policy. Looking at Table 2 and Figure 3, HalfCheetah returns range from −500 to −200, Hopper from 0 to 100, and Walker2d from 0 to 50. These are returns that random policies achieve while falling down or moving backwards in tasks where maximal forward velocity is needed, whereas typical expert behavior policies used for the Minari datasets achieve returns of about 12k for HalfCheetah, 3k for Hopper and 5k for Walker2d. So the claim that “the gate improves 20 of 27 models” mostly compares agents that do not work, with and without the gate, which is one of their main contributions. A behavior-cloning baseline on the expert tier would probably beat every number in the table, but it is not included. The largest performance increase is achieved only in CartPole Swingup, a low-dimensional problem.

  2. The evaluation is too narrow. It has no external baselines, tests only one model class, and uses components that likely do not transfer to other models. The only comparisons are alternative planning penalties on the same MLP ensemble model. Missing are, e.g., behavior cloning, offline MBRL such as MOPO[2] , and model-free offline RL such as IQL [3]. Only one model class is used. Everything is a PETS-style MLP ensemble. The paper is framed around “world models” and cites Dreamer and TD-MPC2, but the method is applied to neither of them, nor to any latent model. There is limited scalability of their approach, which stops the application of their ideas to other world models than MLP ensembles. The support score relies on nearest-neighbor distances in raw standardized state space and on local action-density estimates. Both degrade in high dimensions, and the largest observation space for the continuous tasks here is only 17-dimensional. Furthermore, the pairwise loss requires restoring the simulator to an identical snapshot. This is impossible for offline datasets (which is why their Minari results are gate-only).

[1] N. Lambert, B. Amos, O. Yadan, and R. Calandra, "Objective Mismatch in Model-based Reinforcement Learning", 2020.

[2] T. Yu, G. Thomas, L. Yu, S. Ermon, J. Zou, S. Levine, C. Finn, and T. Ma, "MOPO: Model-based Offline Policy Optimization", 2020.

[3] I. Kostrikov, A. Nair, and S. Levine, "Offline Reinforcement Learning with Implicit Q-Learning", 2022.

Questions for authors

I do not have specific questions that I consider likely changing my assessment, which is based on the above-mentioned critical weaknesses. IMHO, these require a full revision and are not easily addressed through answers.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.