- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper argues that total world-model prediction error is not a reliable measure of planning quality. The authors test this claim in an key-door-goal gridworld with six relevant and six distractor state dimensions. They create synthetic world models by adding noise to the true dynamics: controls noise on relevant dimensions and controls distractor noise. The authors compare models with the same total error, . The proposed Decision-Relevant Prediction Error (DRPE) measures rollout error only on relevant dimensions. Across 48 controlled and 7 learned models, DRPE correlates more strongly with planning success than ( vs. ).
Critical strengths
None
Critical weaknesses
-
W1: The headline correlation holds by construction.
- The planner uses predictions only through the relevant dimensions (Section 3). Irrelevant noise therefore cannot affect its decision (Proposition 1). The experiments seem to only show the lack of planner's access to the irrelevant noise (expected result).
- Since irrelevant noise does nothing (previous point), the 48 controlled models practically only provide 8 data points. Success is constant within each row (Section 5.1, Figure 2a), so is calculated over 8 values, each repeated six times; hence, this is not a clean experimental design.
- Total error looks weak because the sweep introduces large irrelevant error. reaches 1.2, whereas the learned models in Table 3 have between 0.10 and 0.36. So this seems to reflect the "extreme" settings chosen by the authors.
- The authors did not try a more realistic case where the agents are not blind to the dimensions. Dreamer-style agents read reward and value from the full latent state. A planner that reads the full predicted state would test whether DRPE still beats .
- All results come from one toy world (12-dimensional gridworld).
-
W2: Outside the controlled family, DRPE does not rank models correctly.
- On the 7 learned models, DRPE has the wrong sign (, Section 5.2). The rel-only model has the lowest DRPE (0.011) but the worst success rate (0.04, Table 3).
- DRPE needs a known relevance mask. The authors' estimated weights fail (Limitations).
- The abstract's pooled comes from the controlled models. Section 5.2 says so, but the abstract does not.
-
W3: The novelty is overstated.
- Lower prediction error better control is the "objective mismatch" problem of Lambert et al. (2020), which is not cited. Weighting dimensions by value sensitivity (Eq. 1) is also close to Farahmand et al. (2017) and Voelcker et al. (2022). The claim that "the iso-error construction supplies the missing causal evidence" (Section 2) is too strong.
- The propositions restate the setup. Proposition 1 follows from the planner design. Proposition 2 follows from Proposition 1 (Section 4.3). Proposition 3 says rounding absorbs sub-cell error.
References
- Lambert et al. (2020), Objective Mismatch in Model-based Reinforcement Learning.
- Farahmand et al. (2017), Value-Aware Loss Function for Model-based Reinforcement Learning.
- Voelcker et al. (2022), Value Gradient weighted Model-Based Reinforcement Learning.
Questions for authors
None