acceptodds
Under review as a conference paper at ICLR 2027

Planning with Generated Worlds: Calibration-Aware Cross-World Verification

Abstract

Generative world models increasingly expose 3D environments and action-conditioned rollouts to downstream agents for planning. Yet visual realism, geometric fidelity, and rollout consistency do not establish that generated worlds retain task-relevant structure for reliable action. We formulate this gap as generated-world actionability: whether generated worlds support reliable decisions, when their hypotheses and observations provide decision-relevant evidence, and how that evidence should be combined for decision-making. To investigate and address these questions, we operationalize actionability in route-level planning over competing world hypotheses, with a limited verification budget and potentially unreliable observations. On canonical episodes, single-hypothesis commitment and cross-world voting collide in 20% and 24% of cases, respectively, while random and information-gain verification become less safe as they assimilate additional evidence. These results reveal a Verification Paradox: assimilating additional miscalibrated evidence can increase collision risk when correlated world errors and misaligned observations are fused overconfidently. We therefore introduce Calibration-Aware Cross-World Verification (CWV), a training-free decision layer that forms cross-world route beliefs, targets decision-critical verification, down-weights inconsistent observations through innovation-gated inverse-variance fusion, and commits using lower confidence bounds. Base CWV is collision-free across canonical camera budgets; Gated CWV removes the matched-seed high-budget negative-control failure and reduces regret by 66%. Broader evaluations identify three empirical boundaries of actionability: candidate informativeness, task sensitivity, and observation calibration. Together, these results show that generated worlds can support reliable action when decision-relevant variation and external evidence are used in a calibrated way.

Primary area:
applications to robotics, autonomy, planning
Keywords:
world models, planning under uncertainty, agents, uncertainty calibration, decision making

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_59ECOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies that visual and geometric quality metrics for generated worlds do not necessarily reflect their reliability for downstream task planning. The authors identify a Verification Paradox, in which assimilating additional miscalibrated evidence can increase collision risk and worsen downstream performance. To address this, they propose Calibration-Aware Cross-World Verification (CWV), a training-free decision layer that forms cross-world route beliefs, prioritises task-critical verification, and combines inverse-variance fusion with lower-confidence-bound commitment. An innovation gate down-weights observations that are inconsistent with the cross-world belief. The method is then evaluated across multiple environments.

Critical strengths

1 The paper is well motivated and addresses an important practical problem: the gap between upstream generative world model performance and downstream task-specific decision-making.

2 The experiment is solid, with many statistical results supporting the reliability of the results.

Critical weaknesses
  1. The author discusses one of the key reasons for the Verification Paradox: shared structural hallucinations in cross-world agreement, but does not address the problem. Adding sigma_0 (Eq. 9) increases uncertainty, while small sigma_r (cross-world agreement)can still deprioritize confident wrong routes (Eq. 13). The gate (Eq. 15) may also down-weight corrective observations, since it does not determine whether the prior or the observation is wrong. Table 3 shows Base CWV worsening as B increases in Manhattan Midtown, Phoenix Suburb, and Seattle Downtown, raising doubts about whether the problem has been addressed.

  2. The writing and figure presentation make the paper difficult to read. Simple transformations introduce too many symbols, and sigma_0 and sigma_o are easily confused. Fig. 3 is too small to read without zooming; some legend entries are difficult to distinguish, and some curves lack explanations. Also, I could not find in-text references to Figs. 2 and 5 or understand where they fit into the discussion. These issues made it difficult to understand the method and results.

Questions for authors
  1. Does CWV really resolve the Verification Paradox? If cross-world agreement is from shared structural hallucinations, a conflicting observation may be correct. But the gate always down-weights that observation without identifying which source is wrong. Will this strengthen the error agreement? Table 3 also shows Base CWV’s collision rate increasing with the verification budget in several regions. Gated CWV also does not solve this problem(Eq. 15).

2 In the top-left subplot of Fig. 1 left, what do the colored curves, the black curve, and the red dashed line mean? Also, the left figure’s legend does not distinguish solid and dashed lines. The colors of Base CWV and InfoGain Robust in Fig. 2 are also difficult to distinguish. Also, please address Fig. 3’s size and the missing in-text references to Fig. 2 and Fig. 5. 3 What exactly is the route cost function mentioned around line 185? Also, why is TC-EMA included? I could not find an explanation for why this baseline is introduced near Tab.1/ in the main text.

4 Is the second paragraph of the Discussion discussing that upstream generation and downstream decisions should be more closely connected? Appendix H feels disconnected from the paper’s topic. How does it relate to the specific components of CWV?

Reviewer_4qXWOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This work asks whether an agent can make reliable decisions using generated environments. It proposes Calibration-Aware Cross-World Verification (CWV), which combines multiple imperfect world hypotheses before selecting a route. The paper identifies a phenomenon: a fixed verification policy becomes less safe as it assimilates more evidence. The proposed Calibration-Aware Cross-World Verification (CWV) can improve this by aggregating route-clearance predictions across worlds, prioritizing decision-critical observations, fusing evidence, and selecting routes using conservative confidence bounds. In controlled simulations, CWV improves the safety–efficiency tradeoff over several baselines, while observation gating reduces failures under deliberately misleading evidence.

Critical strengths

The paper's motivation is useful - instead of visual fidelity, it evaluates generated worlds by whether they support safe downstream decisions.

The proposed method is training-free, though I would not say it is fast. Inferencing world predictions multiple times can be very challenging and unlikely to be real time.

Ablations show that targeting decision-critical routes improves efficiency. Broader experiments expose limitations from shared world errors and unreliable observations.

Critical weaknesses
  1. Methodology contribution is unclear. This work combines heuristic observation selection, Gaussian evidence fusion, outlier gating, and confidence-bound decisions; it is however unclear what is the novel insight.
  2. It is unclear why world model is needed if the only use is to compute route-clearance means and variances. The problem closely resembles route selection over an ensemble of uncertain maps.
  3. The “Verification Paradox” is unconvincing. Surely additional observations can worsen decisions when they are unreliable and incorporated overconfidently. It is unclear to me why this is a paradox. CWV downweights them, but I would consider the baselines to be too naive.
Questions for authors

Please respond to the weakness I listed, particularly point 2.

Reviewer_7sBSOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether generated worlds provide useful evidence for downstream planning. It considers choosing among candidate routes when the available world hypotheses may share errors and only a few local verification queries are allowed. The experiments show cases where additional verification hurts fixed planning strategies when observations are misaligned or trusted too strongly. The paper evaluates the approach through controlled geometric perturbations, geographic variations, and several representation extensions.

Critical strengths
  1. The paper asks a useful question. A generated environment can look convincing while still giving an agent the wrong reason to choose a route. Evaluating the consequences of those errors, rather than just reconstruction quality, is a worthwhile direction.
  2. CWV is a practical decision layer that does not require retraining the world model. Its components are easy to understand, and the separation between choosing observations, updating beliefs, and making the final commitment makes the approach reasonably easy to adapt.
  3. The paper makes a substantial effort to examine when the approach works and when it fails. The fusion/commitment ablations are useful, and I appreciate the discussion of shared-bias failures in Chicago and uninformative candidates in Elevate3D. These cases give a more helpful picture of the method than reporting only the successful Berkeley setting would.
Critical weaknesses
  1. I am not yet convinced that the main experiments capture the difficulty of planning with actual generated-world errors. In the canonical setup, the reference is itself a generated mesh, and the competing hypotheses are constructed by perturbing it. The negative control deliberately assigns observations to the wrong route. These are useful controlled tests, but the resulting failures are also fairly expected when a method directly replaces its belief with an unreliable observation. The geographic and cross-source experiments broaden the evaluation, but the measured LiDAR case is permissive enough that every method has zero collisions. This leaves a gap between the motivating problem and the strongest experimental evidence.
  2. It is hard to tell how much of the advantage comes from the proposed task-critical reasoning, versus standard conservative updating and outlier handling. The paper includes useful component ablations, but the strongest negative-control result comes from adding the innovation gate. I would like to know whether adding the same gate to InfoGain Robust, or using a simple robust observation update, would recover most of that benefit.
  3. Some reported statistics need to be reconciled before I can rely on the strength of the empirical claims. For example, Table 1 gives zero CWV collisions and five Random Verify collisions over 50 camera episodes at B=2. The stated continuity-corrected McNemar test gives approximately p=0.074, whereas Table S7 reports p=0.0234. Figure S16 also says n=30 per cell, although its cell annotations appear to divide 30 samples across the entire grid. Could you explain why or this may be my mistakes.
Questions for authors
  1. Which existing experiment provides the strongest evidence on independently generated hypotheses evaluated against a measured reference in a setting where some routes are genuinely unsafe?
  2. Have you evaluated InfoGain Robust with the same innovation gate used by Gated CWV? How much of the collision and regret improvement remains attributable to task-critical query selection under that comparison?
  3. Can you reconcile the Table S7 McNemar results with the collision counts in Table 1, and clarify the actual sample allocation in Figure S16? Please indicate whether correcting these changes any of the reported conclusions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.