- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates the rationale behind the LeWM’s struggles under long-horizon scenarios. Main argument of the investigation is that the accumulated errors during the rollout is not the main bottleneck of this struggle, and rather the metric (L2 distance) that evaluates the action sequence candidates is very misleading.
Critical strengths
- The way authors tackle the evaluation protocol of the LeWM makes a good sense. The edge cases authors insisted; tasks solved at start, number of tasks, held-out episodes all reveal vulnerability of the current protocol, and sound critical.
- Cost being more critical factor than the rollout error makes sense, and many studies to add or modify the L2 cost of the planner support the claim as well.
- Table 8 that tries to ablate the effect of rollout error and erroneous cost is helpful, although in appendix.
Critical weaknesses
- Paper may be undone at this stage: L104: appendix zyz, L130: appendix abc, L172: (cite), L215: see appendix and also cite that point this out, L432: (a,b,c,d,e,g,…), L433: Appendix ??, … etc.
- To make this paper’s claim firm, it would have been better if authors come up with better benchmark that covers the aforementioned vulnerability of the current protocol, not insisting that PointMaze-giant is the only environment that suffices the conditions that authors’ checklist. PointMaze cannot cover diverse scenarios that world model has to handle, and analysis only on it may be constrained to a single environment.
- From Table 2, using true dynamics also gives Push-T and Cube-play a huge improvement over each counterpart. The author’s claim may come from the easier action space that PointMaze has: it’s an environment where the 2D point never interacts with surroundings, and it just takes velocity command.
Questions for authors
- Fig.3 demonstrates that increasing the lookahead results in performance degradation, and this effect is confounded with search difficulty and accumulated rollout error. What happens if you sweep this lookahead range under true dynamics? Solely attributing its degradation to CEM’s sample efficiency (Pinneri et al., 2020) is not sufficient.
- The learned cost-to-go is learned by TD on the demonstrator’s own trajectories. The gain obtained from using the cost-to-go to make the demonstration go over similar trajectory pattern that it has shown in the training dataset. Can the authors evaluate the cost-to-go condition on goals not tied to the same-episode construction (the random-goal mode of stable-worldmodel (Maes et al., 2026), or OGBench’s official PointMaze-giant tasks) and report whether the cost’s advantage over L2 (and over the perfect predictor) persists there?