acceptodds
Under review as a conference paper at ICLR 2027

JEPA-UQ: Benchmarking Uncertainty in Latent World Model Rollouts

Abstract

Latent world model rollouts need uncertainty estimates tied to future prediction error. JEPA-UQ defines a measurement protocol that specifies prediction-time information, target loss, horizon, output meaning, and statistical unit. We study one frozen visual encoder and two variance-based predictive systems on 14 nuScenes logs outside the project's recorded history. Both systems show a positive scoreloss association at H1-H5, with weaker correlations at longer horizons. At H1 and H2, AUROC intervals lie above chance. Overall, score rankings carry information about continuous rollout loss across all five horizons. Interval evidence for thresholded failure events is concentrated at H1 and H2.

Primary area:
datasets and benchmarks
Keywords:
Joint-Embedding Predictive Architecture, Latent World Models, Uncertainty Quantification, Rollout Uncertainty, Predictive Reliability, Autonomous Driving

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_gyDxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents JEPA-UQ, a measurement protocol designed to assess uncertainty in latent world-model rollouts. The protocol asks that uncertainty claims include four distinct components and separates such claims into three tiers. The empirical study evaluates Level A directly, investigates Level B, and treats Level C as an extension to the protocol. The protocol is well considered and the experiment is rigorously checked, but the empirical coverage is still too limited to support benchmark-level claims. Expanding to more encoders and datasets, adding a fixed-predictor track, including more rare events, and performing downstream validation would significantly strengthen the submission. The paper is likely to be at the borderline, I lean towards "weak rejection" but would be happy with "weak acceptance" as well if other reviewers suggest so.

Critical strengths
  1. The paper presents a well-defined framework that differentiates between ordinal risk scores, calibrated probabilities, and joint predictive distributions.

  2. Its clear distinction between training, development, calibration, and evaluation datasets is outstanding. It also appropriately manages overlapping windows through scene-level clustered resampling.

  3. It clearly discloses horizon-specific findings, counts of rare events, limits on uncertainty, and any corrections made after the freeze. The conclusions are suitably cautious: a continuous uncertainty ranking is still informative, but failure detection is mainly supported at shorter horizons.

Critical weaknesses
  1. It assesses just one encoder, a single predictor family, one dataset, one camera perspective, and two closely related variance-based uncertainty approaches.

  2. Because the ensemble and MC-dropout methods produce different predictions and failure labels, the study cannot disentangle the quality of the uncertainty estimator itself.

  3. Evaluation at longer time horizons is statistically underpowered, with just 6–20 failure events. Level B lacks strong validation, and Level C has not been deployed. Latent MSE is not tied to any physical or decision-relevant downstream endpoint.

Questions for authors

Do the findings remain consistent across different calibration quantiles?

In what way does latent MSE correspond to physical or semantic prediction error?

Reviewer_h4RQOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies whether uncertainty scores from latent world models can tell, before the future is observed, which rollouts will later incur large prediction errors. The authors propose JEPA-UQ, which is an evaluation protocol rather than a new uncertainty method. The protocol defines uncertainty as a prediction-time statement about a specific future loss, and it fixes the information available at prediction time, the target loss, the horizon, and the statistical unit of evaluation. It also distinguishes three levels of claims that an uncertainty output can make, namely a ranking of risk, a calibrated numerical risk such as a failure probability, and a full distribution over future latent trajectories.

The validation is a single case study on nuScenes front-camera video, using a frozen V-JEPA-style encoder and an autoregressive latent predictor that rolls out several steps into the future. Two uncertainty systems are compared, a deep ensemble and MC dropout, and both score the spread of their predictions. All models, thresholds and calibration maps are fixed on earlier splits and then transferred without retraining to nuScenes logs that the project had not used before. The scores are positively correlated in rank with the realized latent error at every horizon, although the correlation weakens as the horizon grows, while detection of large failures is better than chance only at short horizons. The probability level fails at longer horizons because the calibration maps collapse to constant outputs, and the trajectory distribution level is defined but never tested.

Critical strengths

1. Clear definition of what an uncertainty score should predict

  • Ties each uncertainty output to a specific future loss, a fixed horizon, and the information available at prediction time.
  • Separates three kinds of claims: risk ranking, calibrated probability, and full trajectory distribution.
  • Why it matters: uncertainty in world models is often reported as raw prediction spread without stating what it should predict. The paper's own results show why the distinction is needed, since the scores rank errors reasonably while the derived probabilities fail at longer horizons.
  • This framing is the main reason I find the paper worth considering.

2. Careful and transparent evaluation design

  • Models, thresholds, and calibration maps are fixed before testing and transferred to unseen logs without retraining.
  • Intervals are computed by resampling whole scenes rather than overlapping windows.
Critical weaknesses

1. The protocol itself is not validated

  • No ablation of the protocol components, such as rollout vs teacher-forced evaluation, scene-level vs window-level resampling, or frozen vs test-fitted thresholds.
  • As a result, it is unclear what each design choice contributes, or whether the protocol changes any conclusion compared with standard selective prediction evaluation.
  • Level C is defined but never tested, and the formal targets are not used to derive any new estimator, metric, or theoretical result.

2. Narrow empirical scope

  • One encoder, one dataset, and two standard uncertainty methods.
  • Failure events at longer horizons are too few to support conclusions, and some cells cannot be reported at all.

3. The experimental section is hard to follow

  • Undefined terms
    • "permutation intervals" (Sec. 6.2, L394) are never defined, while all other intervals in the paper come from the bootstrap.
    • "same-label companion check", "label-robust", and "decisive" (Sec. 6.2, L411–413; also App. B.3, L740) are not explained. For example, it is unclear what "H4 is decisive under MC-dropout labels alone" means, especially since ensemble H4 is N/E.
  • Unexplained multiple-comparison family
    • "the audited family of 35 paired comparisons" (Sec. 6.2, L408) is used for Holm correction, but which comparisons it contains is not stated.
  • Unclear statements
    • "This is the first cross-horizon signal we observe" (Sec. 6.2, L379) does not say what it is compared against.
  • Repetition
    • OFR20 retention is reported twice in consecutive paragraphs in different units (Sec. 6.2, L403–406 and L415–418).
    • "Calibration used 86 windows from three logs, with approximately 7–9 positive events" is repeated almost verbatim in Sec. 6.2 (L420), Sec. 7 (L440), and App. A.1 (L574).
  • Process history mixed into the main text
    • The post hoc reporting floor (Sec. 4.4, L236–240) and the OFR20 bug fix (Sec. 6.1, L294–299) interrupt the method and results sections.
  • Broken references
    • "Appendix Table eftab:v3-preliminary" (Sec. 6.1, L291) and "Appendix efapp:ofr-repair" (Sec. 6.1, L299).
  • Overall, Sec. 6.2 reads as a dense list of numbers and caveats rather than a coherent narrative, which makes it very difficult to identify the main findings.
Questions for authors

Clarification requests

  • What do "permutation intervals", "same-label companion check", "label-robust", and "decisive" mean (Sec. 6.2, L394 and L411–413)? In particular, what does "H4 is decisive under MC-dropout labels alone" mean, given that ensemble H4 is N/E?
  • Which comparisons make up "the audited family of 35 paired comparisons" (Sec. 6.2, L408)?
  • What is "This is the first cross-horizon signal we observe" (Sec. 6.2, L379) compared against?
  • The failure thresholds are set so that about 10% of calibration windows fail, but the failure rate on the test logs is much lower at longer horizons. Is this due to the small calibration set or to a distribution difference between the calibration and test logs? Does the chance-level failure detection at longer horizons reflect a real limitation of the uncertainty scores or simply too few failure events?
  • Do the formal targets or Levels B and C lead to any new estimator, metric, or theoretical result beyond what is already used in standard uncertainty evaluation?

Requested additional experiments

  • An ablation of the protocol components showing whether the conclusions or the ranking between methods change under teacher-forced instead of rollout evaluation, window-level instead of scene-level resampling, and test-fitted instead of frozen thresholds.
  • Results with additional encoders, datasets, and uncertainty method families, to show that the protocol is useful beyond a single setting.
  • An empirical instantiation of Level C.
  • A fixed-predictor comparison that holds point predictions constant, so that differences between uncertainty estimators can be separated from differences in predictive accuracy.
Reviewer_qyFBOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents an evaluation protocol called JEPA-UQ, for evaluating uncertainty estimates in latent world model rollouts. The authors evaluate two existing uncertainty estimation approaches - deep ensembles and MC dropout, using a frozen visual encoder and camera-only latent predictor. The results show positive correlations between uncertainty scores and realized prediction errors across all five horizons, although the correlations weaken as the prediction horizon increases. Failure-detection AUROC confidence intervals are above chance only at the first two horizons. Numerical risk calibration remains unvalidated, and joint trajectory uncertainty is not empirically evaluated. Overall, the paper contributes a structured evaluation protocol and an empirical characterization of the relationship between predictive disagreement and future latent prediction error in a limited world-model setting.

Critical strengths

Careful empirical evaluation with explicit limitations. The authors conduct a horizon-resolved evaluation on 14 logs outside the project's recorded experimental history, using frozen models and calibration artifacts. The results reveal a useful distinction between predicting continuous rollout error and detecting large-error events, particularly as prediction horizons increase. The authors also acknowledge the limitations of their calibration results and experimental scope, which strengthens the credibility of their empirical conclusions.

Critical weaknesses
  1. Limited novelty and unclear methodological contribution. The proposed JEPA-UQ framework primarily organizes established concepts from uncertainty quantification, including uncertainty ranking, probability calibration, and selective prediction, into a horizon-specific evaluation protocol. The paper does not introduce a new uncertainty estimation method, theoretical result, or evaluation metric, nor does it clearly demonstrate which limitations of existing evaluation approaches necessitate a new framework. Consequently, the paper's contribution appears incremental, and its distinction from existing work is insufficiently established.

  2. Narrow empirical validation relative to the scope of the proposed framework. The evaluation is restricted to one frozen visual encoder, one dataset, and two established variance-based uncertainty approaches. Furthermore using their proposed terminology, only Level A (uncertainty ranking) is directly validated, while Level B (numerical risk estimation) remains unvalidated and Level C (joint predictive distributions) is not evaluated. These limitations make it difficult to assess the framework's generality or establish substantive conclusions about uncertainty estimation in latent world models beyond this specific experimental setting.

Questions for authors
  1. Could the authors clarify which specific methodological limitations of existing uncertainty evaluation frameworks JEPA-UQ addresses? In particular, what aspects of recursive latent rollouts require a new protocol rather than applying existing metrics independently across prediction horizons? A concrete example demonstrating a difference in evaluation conclusions would help establish the contribution's significance.

  2. The authors state that "Holm-adjusted H1 comparisons favor ensemble and MC dropout on ranking and nAURC." However, these favorable comparisons appear concentrated at H1, while the evidence at longer horizons is weaker. Could the authors clarify what the exploratory baseline comparisons establish about the advantages of ensemble disagreement and MC dropout over simpler uncertainty estimators?

  3. The authors report that "The frozen OFR20 cutoffs retain 0.0592–0.0822 of the new windows across the two UQ systems." Given that these cutoffs are determined from the 20th percentile of calibration scores, could the authors explain this substantial shift in retained coverage? What does this imply about the stability and practical usefulness of the uncertainty scores when transferred to unseen data?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.