- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper conducted three sanity checks for world model planning, including: (1) whether changing the plan affects the imagined score more than changing the sampling seed; (2) whether the imagination is closer to the true future of its own plan than to those of perturbed plans; and (3) whether the VLM evaluator reproduces its own decision under repeated queries with fixed inputs.
Critical strengths
The three checks are important and timely.
Critical weaknesses
- The writing of this paper is very unclear. The overall writing and structure are more like an experiment report than an academic paper.
- The innovation is limited. Even if the results are interesting, the contribution does not seem to meet the quality of top venues like ICLR.
- The quality of the results is doubtful since only one world model and two planners are used.
Questions for authors
- Please consider clarifying why the three sanity checks could support a technical paper and the robustness of the results.
- Why did the authors choose to ask the three questions for the sanity check? What is their relationship to and meaning for future research?
3 comments
Thank you for your response. I increased my score to 2.
Thank you for reading our response and updating your assessment. In case it is useful alongside your original questions on technical scope and practical use, the latest PDF adds two brief results prompted by our discussion with the other reviewers: Corollary 1 and Appendix J (Table 8), which show that Check 1 () tracks single-rollout agreement with true simulator rankings across both models and channels (from 49% to 68%, rank correlation 0.94), and "Using the checks" in Section 6, which turns Checks 1-3 into a step-by-step pre-deployment decision rule. Thank you again for your time and feedback.
Response to Reviewer 7TFj
Thank you for finding the three checks important and timely, and for the direct assessment of the presentation and the contribution. The revision restructures the paper around one question, whether the imagined outcome depends on the plan being scored, and adds theory and experiments that address the contribution and robustness concerns.
Q2 (why these three questions, and what they mean for future work). Planning with a world model chains three components: the generator must produce futures that depend on the plan, those futures must match what the plan would actually do, and the evaluator that compares them must return a stable choice. A failure at any link makes the planner rank something other than its plans, and none of them is visible in success rates: on World-in-World's 144 navigation episodes, an image-to-video model that receives only the initial heading of a plan (40.3%) matches a trajectory-conditioned navigation model (40.3%). Each check tests one link against a reference scale that the system itself supplies (sampler seeds, the simulator's futures of alternative plans, re-queries of the evaluator), which is why they apply to any generator without training. Section 1 now states this argument. Section 6 gives the consequences for future work: report the ratio per conditioning channel, budget rollouts with Proposition 1, calibrate fidelity against the true-future effect, and test confidence signals at the step level with scene and episode length fixed.
Q1 and W2 (what supports a technical paper). Four results go beyond measurement. (i) Proposition 1 shows that the plan-to-seed ratio bounds the expected-score gap between neighboring plans, ∣μ(a′)−μ(a)∣≤2 σR for any score distribution, and, under Gaussian noise, the probability that a planner comparing K rollouts ranks them correctly, Φ(2K/π R); at the measured ratios a single rollout is correct at most 74% (VLM planner) and 68% (heuristic planner) of the time. (ii) The channel decomposition locates the failure: in the same model, run and seeds, perturbations that only rotate the generator's input move the score 1.92 times as much as reseeding, while perturbations conveyed only by the text prompt move it 0.32 times as much. This is also a positive control showing that Check 1 discriminates. (iii) The calibrated Check 2 shows that every perturbation turns the true future's final view by 22.5 or 45 degrees and that the re-rendered true futures end 0.62 LPIPS apart, while the imagination recovers 1.6% of that distance. (iv) The within-episode analysis shows that the imagined score predicts the progress of each step (partial correlation 0.19, 95% CI [0.05, 0.32]) while the confidence signals proposed for world models do not, and that the oracle's episode-level inversion comes from differences between episodes.
W3 (one world model and two planners). The revision adds a second generator. The benchmark also provides LTX-Video 0.9.1, a separately trained image-to-video model with its own autoencoder and text encoder. With LTX in place of Wan2.2 and the same view rotation, prompts, seeds and scorer, we regenerated all eight rollouts of all 326 VLM-planner steps. The channel split reappears: text-only perturbations move the LTX score 0.27 times as much as reseeding (95% CI [0.22, 0.33]) and perturbations that change the input view move it 2.42 times as much ([2.05, 2.83]; 2.60 when only the view changes), against 0.32 and 1.48 (1.92) for Wan2.2 on the same steps. On the final frame the LTX ratio is 0.75 ([0.64, 0.87]). On Check 2 the LTX imagination recovers 2.8% of the true-future distance (95% CI [0.5, 5.2]), Wan2.2 1.6% on the same steps. On 32 conditioning frames offline (Figure 2 shows one), writing three turns instead of three forward steps moves LTX's final frame 0.54 times as much as reseeding ([0.39, 0.72]), while the same turns applied as a rotation of the input move it 1.58 times as much; for Wan2.2 the two ratios are 0.61 and 1.05 (Section 5.1, Appendix E). Inside each system the same generator passes Check 1 through one channel and fails it through the other, and Check 2 is calibrated against the simulator's exact futures, re-rendered for all 1,304 compared plans. Robustness is reported with episode-clustered intervals throughout; the main ratios exclude 1 and the plan effect is smaller than the seed effect in 20 of 23 and 10 of 10 episodes.
W1 (writing and structure). The introduction now opens with the problem and the evidence that success rates cannot reveal it, then motivates one check per link of the planning chain and states the proposition and the findings. Section 4 defines each check by a question, a reference scale and its decision consequence, and Section 5 leads each subsection with its finding. Descriptive statistics that did not support a conclusion were moved out of the main text. The title is unchanged; the abstract was revised to state the results directly.