acceptodds
Under review as a conference paper at ICLR 2027

Not All Errors Matter: Decision-Relevant Prediction Error Predicts Planning Quality

Abstract

World models are typically trained and evaluated by prediction error, assuming that more accurate predictions lead to better decisions. We show that this assumption can fail because models with similar total error can differ substantially in planning performance when their errors occur on different state dimensions. We introduce Decision-Relevant Prediction Error (DRPE), which measures prediction error on the state dimensions that affect decisions. We also develop an iso-error evaluation protocol that varies error allocation while keeping total error fixed. In a factored gridworld with known state relevance and a standardized planner, we evaluate 55 controlled and learned models across different error levels and allocations. Total prediction error is weakly related to planning success (Spearman ), whereas DRPE is strongly predictive (; within the controlled family). Models with only a 1% difference in total error can differ by 60 percentage points in planning success (97% vs 37%). The relevant error also depends on the task, with model rankings reversing across tasks at the same total error. Deeper imagination further amplifies decision-relevant errors, while learned models exhibit systematic bias on rare but decision-critical events. We formalize sufficient conditions under which DRPE correctly ranks models and total prediction error cannot.

Primary area:
reinforcement learning
Keywords:
World models, Decision-Relevant Prediction Error, Model-based Planning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_V6QuOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper argues that total world-model prediction error is not a reliable measure of planning quality. The authors test this claim in an 8×88 \times 8 key-door-goal gridworld with six relevant and six distractor state dimensions. They create synthetic world models by adding noise to the true dynamics: σR\sigma_R controls noise on relevant dimensions and σI\sigma_I controls distractor noise. The authors compare models with the same total error, EtotE_{\text{tot}}. The proposed Decision-Relevant Prediction Error (DRPE) measures rollout error only on relevant dimensions. Across 48 controlled and 7 learned models, DRPE correlates more strongly with planning success than EtotE_{\text{tot}} (ρ=−0.84\rho = -0.84 vs. −0.25-0.25).

Critical strengths

None

Critical weaknesses
  • W1: The headline correlation holds by construction.

    • The planner uses predictions only through the relevant dimensions (Section 3). Irrelevant noise therefore cannot affect its decision (Proposition 1). The experiments seem to only show the lack of planner's access to the irrelevant noise (expected result).
    • Since irrelevant noise does nothing (previous point), the 48 controlled models practically only provide 8 data points. Success is constant within each σR\sigma_R row (Section 5.1, Figure 2a), so ρ=−0.98\rho = -0.98 is calculated over 8 σR\sigma_R values, each repeated six times; hence, this is not a clean experimental design.
    • Total error looks weak because the sweep introduces large irrelevant error. σI\sigma_I reaches 1.2, whereas the learned models in Table 3 have EtotE_{\text{tot}} between 0.10 and 0.36. So this seems to reflect the "extreme" settings chosen by the authors.
    • The authors did not try a more realistic case where the agents are not blind to the dimensions. Dreamer-style agents read reward and value from the full latent state. A planner that reads the full predicted state would test whether DRPE still beats EtotE_{\text{tot}}.
    • All results come from one toy world (12-dimensional gridworld).
  • W2: Outside the controlled family, DRPE does not rank models correctly.

    • On the 7 learned models, DRPE has the wrong sign (ρ=+0.43\rho = +0.43, Section 5.2). The rel-only model has the lowest DRPE (0.011) but the worst success rate (0.04, Table 3).
    • DRPE needs a known relevance mask. The authors' estimated weights fail (Limitations).
    • The abstract's pooled ρ=−0.84\rho = -0.84 comes from the controlled models. Section 5.2 says so, but the abstract does not.
  • W3: The novelty is overstated.

    • Lower prediction error ≠\neq better control is the "objective mismatch" problem of Lambert et al. (2020), which is not cited. Weighting dimensions by value sensitivity (Eq. 1) is also close to Farahmand et al. (2017) and Voelcker et al. (2022). The claim that "the iso-error construction supplies the missing causal evidence" (Section 2) is too strong.
    • The propositions restate the setup. Proposition 1 follows from the planner design. Proposition 2 follows from Proposition 1 (Section 4.3). Proposition 3 says rounding absorbs sub-cell error.

References

  • Lambert et al. (2020), Objective Mismatch in Model-based Reinforcement Learning.
  • Farahmand et al. (2017), Value-Aware Loss Function for Model-based Reinforcement Learning.
  • Voelcker et al. (2022), Value Gradient weighted Model-Based Reinforcement Learning.
Questions for authors

None

Reviewer_sNezOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper is centrally concerned with the question: "Does prediction accuracy measure the right thing for evaluating a world model?" The authors seek to address this question by separating so-called "error magnitude" and "error allocation". They ask whether models with the same total prediction error can nevertheless support different decisions, given different distributions across states. To answer this, the authors construct a controlled gridworld environment, and find that "decision-relevant predictor error" (DRPE) correlates with success more strongly than does total error alone. The give three propositions that formalize their results.

Critical strengths

This paper is exceptionally clear and well-written, and the experimental set-up is clean. The iso-error protocol, in particular, is nice. The paper is also upfront about its limitations.

Critical weaknesses

My principal concern is that the authors' key results seem to beg the question. In particular, so far as I can see, the planner only looks at the rounded relevant coordinates of a prediction, and then plugs them into the relevant V*. In turn, this seems to mean that noise on the irrelevant dimensions can't affect any decision. That is what the authors call Proposition 1. After that, however, the controlled models inject noise, with Table 6 showing the upshot. But the correlations seem to me to mostly restate the design---something that the authors more-or-less concede in the Appendix, when they say that the central claim is "true by construction". On its own, I think, that wouldn't be so bad. But on the seven learned models, it seems that the one test not set up to favor DRPE is such that it does no better than total error. So, the upshot seems to be that the relevant-only model has the lowest DRPE score, but the worst success overall. Is that right?

Questions for authors

Can the authors assuage the worry I just outlined in the critical weaknesses section?

Reviewer_K3UaOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies how the allocation of world-model prediction error affects planning. It introduces Decision-Relevant Prediction Error (DRPE), a weighted multi-step error over task-relevant state dimensions, and an approximately iso-error evaluation protocol. Experiments use a factored gridworld, controlled perturbations of the true dynamics, and a planner with an exact optimal-value heuristic. The paper demonstrates different planning outcomes at nearly identical total error and examines task dependence, search depth, and learned-model failures.

Critical strengths
  1. The controlled experiments clearly demonstrate that total prediction error can conceal consequential differences in error allocation. The comparison between 97% and 37% success at total errors of 0.151 and 0.152 is particularly clear.

  2. The negative results are informative. DRPE fails to rank the learned models, and estimated relevance weights lose the advantage over total error. The key-pickup example illustrates how average errors can miss a decision-critical transition.

  3. The factored environment, per-dimension profiles, and repeated-seed evaluations make the mechanism interpretable. A controlled study can contribute without large-scale benchmarks or state-of-the-art performance.

Critical weaknesses
  1. DRPE's ranking performance is established only within a restricted perturbation family. The pooled correlation of -0.84 combines 48 controlled models with seven learned variants. On the learned variants, DRPE and total error produce identical rankings, both correlating +0.43 with success. Table 3 gives a direct counterexample: the relevant-only model has DRPE 0.011 and success 0.04, while uniform 12k has DRPE 0.383 and success 0.70. The five 20k-transition variants give +0.60 for both metrics, so data budget does not explain the failure.

    The controlled result is weaker than it appears. All relevant coordinates share one noise scale, and Table 6 reports identical DRPE and success across all six σI values for each σR. The 48 models thus contain eight distinct points, and σR alone gives the same -0.98, as does a mask with only the carrying flag (Appendix A). The correlation, and the equivalence of binary and graded weights, show only that success falls with σR, not robustness to heterogeneous relevant errors. Bootstrapping over replicated points also makes the CIs too narrow. Conversely, total error's weak correlation reflects the σI grid: it is -0.98 at σI = 0 and -0.32 on the full grid. The abstract and conclusion should state these limitations.

  2. The pooled evaluation does not hold the measurement distribution or planner fixed, although Section 3 states that every model uses the same planner. Controlled-model errors use uniform-random actions; learned-model errors use a random/expert mixture (Appendix A). The controlled grid uses depth 4, learned variants use depth 2 with injected planning noise (Appendix D), and Task B uses depth 1 (Appendix H). The pooled correlation is therefore weak evidence for a common ranking metric. A comparison on shared held-out trajectories with a common planner is needed.

  3. The theoretical analysis does not establish the advertised ranking guarantee. Propositions 1 and 3 establish invariance when relevant predictions, or their rounded representations, agree. Neither proves that lower DRPE implies better planning. Proposition 1 also requires agreement on the planner's support, whereas DRPE is measured under the behavior policy, so even zero DRPE need not satisfy it. The Section 4.3 corollary asserts correct ranking when error structure is fixed, but this condition is undefined and Appendix B gives no proof.

    Proposition 2 also needs repair. Its construction makes M_B optimal and M_A degraded, opposite to the stated inequality V(pi_M_A) - V(pi_M_B) > C. Monotonicity is unnecessary, since the limit suffices, but the limiting state-independent action distribution is asserted without proof. The proof also assumes the irrelevant-noise model is optimal, yet the oracle succeeds only 97%. These gaps do not undermine the observed matched-error counterexample.

  4. The contribution beyond value-aware modeling needs sharper justification. VAML and value equivalence already distinguish predictive fidelity from decision usefulness. VaGraM (Voelcker et al., ICLR 2022), which is not cited, is especially close: it weights prediction error by value gradients and studies distracting state dimensions. Appendix A finds that value-prediction error matches DRPE's controlled-family correlation (-0.98). The stated advantage, that DRPE needs only a mask, is not demonstrated here: the mask is derived from V* (Eq. 1), and the estimated mask fails.

  5. The setup supports narrower conclusions than several claims suggest. Relevance is known, irrelevant variables cannot affect relevant dynamics, and the planner uses true reward and exact V*. Irrelevant errors are therefore harmless by construction, while the attempted relevance estimator fails.

    The depth experiment changes both rollout length and the number of noisy candidate plans, 4^D. With one noise sample per leaf, maximizing over more candidates favors spuriously optimistic predictions; contrary to Section 5.4, zero-mean noise does not average out under maximization. The experiment therefore does not isolate error accumulation. Finally, Section 5.3 shows a tie in Task A (0.97 versus 0.97) becoming an ordering in Task B (0.82 versus 0.97), not the ranking reversal claimed in the abstract and conclusion.

Questions for authors
  1. What precise assumptions make DRPE order-preserving for planning return, and where is this proved? Is the corollary a theorem or an empirical observation?

  2. Does DRPE predict success when errors vary independently across relevant dimensions or in temporal correlation (e.g., the AR(1) models of Appendix A), on shared trajectories with a common planner?

  3. Is the primary contribution a practical ranking metric or a controlled characterization of evaluation failure? How do the learned-model results and comparison with VaGraM constrain that claim?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.