acceptodds
Withdrawn from ICLR 2027

Calibrated Uncertainty for Long-Horizon Rollouts in Latent World Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Latent world models can imagine long rollouts, but planners need to know when their predictions become unreliable. We show that common uncertainty measures, including ensemble disagreement, latent divergence, and predictive variance, are Lipschitz functions of compact latent states and therefore have horizon-independent ceilings, while rollout error need not. Across two world-model families, ensemble disagreement grows more slowly than reward error and becomes anti-correlated with it on several tasks. We address this mismatch without retraining or ensembles. A linear probe maps a frozen latent state to rollout error and is calibrated by split conformal prediction at each horizon. The resulting intervals achieve finite-sample coverage and track per-rollout error better than standard conformal baselines on nearly every DreamerV3 and TD-MPC2 task. Where error compounds, they are substantially sharper than ensemble-scaled intervals, while matching them when error remains flat. A single-checkpoint probe matches an ensemble-trained probe at of the training cost with a -member ensemble. For trusted-horizon decisions, our intervals accept as many imagined steps as ensemble disagreement while exceeding the error tolerance less often. Coverage can degrade under substantial behavior-policy shifts, requiring recalibration.

Primary area:
learning on time series and dynamical systems
Keywords:
world models, uncertainty quantification, long-horizon rollouts

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_Z3cWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper investigates why latent uncertainty cannot track long-horizon error. The paper provides theoretical arguments on the above question. They also show empirically that the correlation between true reward error and ensemble disagreement reduces as the rollout horizon grows. Across two world models, DreamerV3 (DMC, Atari benchmarks) and TD-MPC2 (DMC benchmarks), the paper provides a detailed empirical analysis on long-horizon error. They propose a linear calibrated probe to predict rollout's reward error.

Critical strengths

The paper asks a fundamental question of why latent uncertainty cannot track long-horizon error. They provide theoretical arguments on this question. Furthermore, the paper proposes a linear probe to predict rollout's reward error. The empirical evaluation is strong, across many task of DMC and Atari benchmarks, on two world models DreamerV3 and TD-MPC2.

Critical weaknesses
  • The correctness of theorem 4 as stated is not clear. The proof in the appendix is not sufficiently detailed to validate its accuracy. As stated in the Appendix, " sup δt = ∞, which is doubtful on DMC tasks with bounded state spaces".

  • Presentation: The paper's presentation can be improved. (For e.g., Figure 1 size). The paper's key takeaways are distribution throughout in many places and hence are difficult to evaluate.

  • Minor corrections: "Two one-hot vectors over C classes differ in at most one position" in proof of Proposition 2. Though two one-hot vectors differ in at most two positions.

Questions for authors
  • Can the authors provide a detailed and expanded proof for Theorem 4 instead of brief version provided in appendix. It is not clear why many claims hold, such as the bound on U_t. The paper states in appendix that the theorem requires "supt δt = ∞, which is doubtful on DMC tasks with bounded state spaces". Why does the true dynamics not being globally stable imply unboundedness? Please expand on the precise meaning of "Ut cannot order trajectories".

  • The quality of the Figure 1 should be improved include size and resolution.

  • Doesn't the linear probe also satisfy properties in Theorem 4?

  • DreamerV3 (H=15) and TD-MPC2 both use much shorter rollout horizon. Perhaps a more interesting study would be for world models where long rollouts are actually used in practice.

Reviewer_JisjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

A linear probe predicts cumulative reward error from frozen world-model latents, and horizon-specific conformal calibration turns the prediction into an upper error bound. The paper evaluates DreamerV3 and TD-MPC2 across many tasks, including distribution shifts and negative planning results. It motivates the approach through a claimed limitation of uncertainty statistics on compact latent spaces.

Critical strengths
  • The empirical investigation is broad and explicit about negative results. In particular, the paper reports that improved error intervals do not improve closed-loop return and that policy changes can break calibration. Distinguishing calibration quality from control performance is useful.
  • A calibrated readout from a single frozen model is practical. The reported interval-quality improvements on compounding-error tasks are potentially valuable independently of the proposed impossibility argument.
Critical weaknesses
  • Theorem 4’s ranking-impossibility conclusion does not follow from boundedness. For nonnegative error delta, U = delta/(1 + delta) is bounded and strictly increasing, so bounded range alone does not preclude exact ordering. Compactness and Lipschitz continuity constrain magnitude, not the ability to rank arbitrarily large errors. The surrounding discussion recognizes this distinction, but the theorem and its Appendix A proof assert the opposite. Empirical saturation cannot establish that missing implication.
  • The finite-sample coverage guarantee is not established for the supplied evaluation procedure. Proposition 5 requires a scale fitted independently of calibration and test episodes, whereas Section 5, Appendix E.2, and method/tune_probe.py select the ridge penalty using calibration performance. In method/probe_conformal.py, preprocessing uses all features before the episode split, and calibration quantiles pool rollout rows within episodes. An episode split prevents direct episode overlap, but does not by itself establish exchangeability of dependent row-level scores. These issues require a justified calibration protocol; they do not prove that every reported empirical coverage result is wrong.
Questions for authors
  1. How can the ranking-impossibility conclusion in Theorem 4 be reconciled with a bounded strictly increasing transformation of error? Please distinguish what follows from compactness from the observed saturation pattern, and identify any additional assumptions needed for an ordering limitation.
  2. Which exact preprocessing, hyperparameter-selection, and calibration procedure produced the main tables? Do the coverage and interval-score gains persist with preprocessing and model selection restricted to training/validation episodes, untouched calibration episodes, and a calibration unit whose exchangeability is explicitly justified?
Reviewer_mNQmOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies uncertainty estimation for long-horizon rollouts in latent world models and proposes an error calibration method combining linear probes with split conformal prediction. The method freezes the world model, uses imagined latent states to learn the absolute error of cumulative reward predictions, then calibrates separately at each rollout horizon to construct adaptive error upper bounds, based on which a trusted rollout horizon is selected. The authors conduct experiments across multiple environment settings. Results show that the proposed method generally distinguishes per-rollout reward error better than stabilized ensemble-scaled conformal intervals and improves interval score on some tasks, but coverage under behavior-policy shifts and practical planning gains remain limited.

Critical strengths
  1. The paper honestly reports the experiment results, including negative ones.
Critical weaknesses
  1. A key theorem in the paper, Theorem 4, has a core implication that does not hold. A bounded and Lipschitz latent statistic can still correctly rank unbounded physical errors. For example, on the latent space [0,1][0,1], setting x(δ)=δ1+δx(\delta)=\frac{\delta}{1+\delta} and U(x)=xU(x)=x satisfies both the boundedness and Lipschitz conditions while preserving the correct ranking for arbitrarily large errors. Therefore, compactness can only constrain the magnitude of the statistic but cannot imply that long-horizon ranking fails. This theorem also contradicts the statement in the Introduction that “a bounded statistic can still rank rollouts by an unbounded error.”

  2. Proposition 5 requires that the fitting of the scale function ss must not reference the calibration/test episodes. However, Appendix E.2 explicitly states that the ridge penalty λ\lambda is selected on the calibration data. If λ\lambda selection and the computation of the final q^h\hat{q}_h use the same calibration set, then the scale function already depends on these labels through model selection. In this case, Proposition 5 does not apply to the actual procedure adopted in the paper, and the claimed finite-sample marginal coverage guarantee lacks theoretical support.

Questions for authors
  1. Could the authors address the issue with Theorem 4? This affects the paper's correctness.
  2. There are issues with figure layout and readability throughout the paper. For example, Figure 1 is very small despite substantial surrounding whitespace.
Reviewer_DgYSOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

Latent world models, which forecast rollouts under latent dynamics, are important for model-based reinforcement learning, and ensembling of such models can be used for quantifying epistemic uncertainty. However, the architecture of standard models like DreamerV3 poses challenges for such uncertainty quantification due to the boundedness of latent representations and smoothness of uncertainty statistics. The authors prove this formally and empirically demonstrate that the correlation between ensemble disagreement and true error decays in the horizon. The work then proposes learning linear models (i.e., probes) of reward uncertainty. Specifically, the authors propose applying Ridge regression to model the multiplicative scale of error of the latent world model's reward prediction against the latent representation and then split conformal calibration at each horizon to bound the true reward statistically. The authors show that their proposed approach achieves higher adaptivity than the ensemble baseline on both DreamerV3 and TD-MPC2 tasks, as well as improved interval scores on DreamerV3 tasks where reward error compounds with the horizon.

Critical strengths

The work identifies a practically relevant problem with latent world models in which ensemble disagreement fails to accurately quantify error at long horizons. It characterizes this problem theoretically using an argument based on the boundedness of the latent representations and smoothness of uncertainty statistics, and it supports this diagnosis with empirical evidence. The authors then develop a simple, well-founded solution that requires no retraining or architectural changes. Specifically, a linear probe of the frozen latent orders the rollouts, and subsequently per-horizon conformal calibration is used to calibrate the magnitude. Finally, the paper presents a thorough and transparent experimental study covering 40 DreamerV3 tasks and 19 TD-MPC2 tasks with comparisons against a large set of baselines. The authors evaluate a range of metrics, including coverage, adaptivity, and interval score, and are clear about where their approach succeeds and where it falls short.

Critical weaknesses

The primary weakness is that the work does not demonstrate a practical benefit for planning, which was the original motivation for the paper. The authors report that every attempt to turn the improved uncertainty estimates into better decisions was unsuccessful. The secondary weakness is organization and writing. The introduction (Sec. 1) is well written, but the experimental results section (Sec. 6) is difficult to follow. Key definitions are introduced in passing within dense paragraphs, and the discussion packs many numerical results into running text. This makes it hard to identify the main findings and distinguish them from auxiliary results.

Questions for authors
  1. Given that every planning use tested thus far was null, can the authors show a setting where the intervals improve a downstream decision? Such a result would substantially strengthen the paper, and I would raise my score accordingly.

  2. Could the authors add a dedicated subsection at the start of Section 6 that defines the evaluation metrics and terms before the results are presented? This would cover marginal coverage, coverage on the hardest third, late-horizon adaptivity, interval score, the criterion for "compounding" vs. "flat" error, and the stabilized ensemble baseline. A brief illustration would also help readers interpret the numbers that follow, such as one task showing how width, error, and coverage relate across horizons.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.