acceptodds
Withdrawn from ICLR 2027

CalibWM: Action-Fidelity Calibrated Uncertainty for Frozen Video World Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Action-conditioned video world models support robot policy evaluation and imagination-based planning, but their training and evaluation focus largely on vi- sual fidelity. Recent benchmarks show that photorealistic rollouts can misrepre- sent the effects of executed actions and depict failures as successes. Uncertainty estimates calibrated to visual correctness inherit this mismatch. We introduce CalibWM, a lightweight probe on a frozen action-conditioned video world model that produces a spatiotemporal uncertainty map aligned with action-fidelity fail- ure, defined by whether a frozen inverse dynamics model can recover the executed action from generated frames. A quantile objective regularizes uncertainty to grow with rollout horizon, while a spatial term localizes the map and an alignment term trains its readout to predict action-fidelity failure. Per-horizon, segment- level split conformal calibration provides finite-sample coverage for aggregated normalized prediction-error scores under exchangeability, with unit-level impli- cations determined by the spatial aggregator. On Bridge and DROID, CalibWM achieves AUROCs of 0.782 and 0.761, respectively, compared with 0.694 and 0.679 for the strongest baseline trained without the action-fidelity label. Its mar- gins over the strongest control trained with that label are +0.020 (clustered 95% CI [0.009, 0.031]) on Bridge and +0.017 ([0.006, 0.028]) on DROID. On a human- annotated optimism-bias subset with uniformly high visual quality and rare fail- ures, CalibWM achieves 0.634 AUPRC against a chance level of 0.09; an oracle visual proxy with access to the ground-truth future reaches 0.106. We establish the coverage result at each fixed horizon. Under ordering consistency, we also prove that the action-fidelity readout has AUROC no lower than a visual proxy computed from the generated rollout alone, provided that the proxy carries no ad- ditional information about action-fidelity failure given the probe’s feature context.

Primary area:
applications to robotics, autonomy, planning
Keywords:
video world models, action fidelity, uncertainty quantification, conformal prediction, spatiotemporal calibration, robot policy evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_2dYAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work introduces CalibWM, a lightweight probe attached to a frozen action-conditioned video world model to produce spatiotemporal uncertainty maps across latent units and prediction horizons. The probe is optimized using a suite of carefully designed loss functions, including horizon-aware regularization, spatial smoothing losses, proper scoring and distillation losses leveraging cached teacher outputs, and an alignment loss for learning a scalar readout that predicts action-fidelity failures, where such failures are identified by whether a fixed inverse dynamics model cannot recover executed actions from synthetic frames. The method incorporates a segment-level split-conformal mechanism to establish finite-sample marginal coverage guarantees for its latent prediction-error regions, and it provides a further theoretical result bounding the readout’s discriminative performance under stated consistency and proxy assumptions. Empirically, CalibWM achieves strong discrimination performance against replay labels, outperforming both the strongest label-free baseline and the best label-using control; it also yields favorable results on a human-annotated optimism-bias subset, maintains near-target segment-level coverage, improves horizon-dependent uncertainty behavior, and supports out-of-distribution detection via its epistemic uncertainty branch.

Critical strengths

The paper has several strengths: it addresses a well-motivated problem, arguing that uncertainty for world models should reflect action outcomes rather than only visual error; it attempts to separate label access from design by introducing two matched label-using controls together with trajectory-clustered conformal intervals; it features rigorous and carefully scoped conformal treatment, where Proposition 1 is correct and the work clearly distinguishes guaranteed from non-guaranteed claims as well as marginal slack from Beta fluctuation; it pre-specifies falsification criteria (Table 3), and horizon regularization delivers a clear width benefit (ρ 0.96 vs 0.61; h=16 width ratio 1.21 vs 2.43); additionally, the method is cheap at inference, requiring only a single forward pass and fewer than 1% of backbone parameters.

Critical weaknesses
  1. Label validity is the key weakness. The primary evaluation labels (simulator replay of recorded actions) never use the generated video. Table 1 may measure sim-replay difficulty rather than world-model action fidelity. κ between weak and replay labels is never reported.

  2. It is unclear how C³-backbone features are obtained for videos from other generators. There is possible τ_a leakage, no CIs, and the visual-quality selection makes the LPIPS comparison weak by construction.

  3. Small design-attributable gain. It is +0.02 over the best label-using control, confounded by object masks and extra ground-truth supervision. Without the label, CalibWM is no better than C³.

  4. Proposition 2 is near-tautological. Assumption 2 presumes Bayes-optimal ranking through a scalar bottleneck. The proposition is stated for the IDM label but evaluated on replay labels, and its falsification test is effectively not at risk.

  5. Coverage result has no demonstrated use. It guarantees only a latent-space tolerance, does not affect the readout, and has no downstream use. Baselines are not conformalized for a matched-width comparison.

  6. Inconsistencies and gaps. Width 0.82 vs 0.88; π changes despite a post-hoc layer; a 14-point coverage drop despite ρ = 0.96; possible overlap between training and calibration splits. A single backbone, an unspecified IDM, no seed variance, a substantial training-time cache cost, and an incomplete manuscript.

Questions for authors

Q1: Does the replay label use the generated rollout at all? What fraction of segments would be labelled failures for a perfect world model? Please report Cohen's κ and an action/context-only "difficulty" baseline.

Q2: How are C³ penultimate features and action embeddings computed for MiraBench videos from other generators? Which segments were used to fit τ_a? Please give positive counts and clustered CIs.

Q3: Do the label-using controls use the same object-mask aggregator, ranking surrogate, weighting, and schedule? Where do masks come from at deployment?

Q4: Which Y does Proposition 2 refer to? Are masks part of Φ? Would you test Assumption 2 against a high-capacity classifier on full Φ? Are probe-training and calibration segments disjoint (§4.1 suggests both use the full cache)?

Q5: Please explain the 0.82/0.88 width discrepancy, the π change under a post-hoc layer, and the coverage collapse of the w/o-conformal variant despite ρ = 0.96. Please define ECE.

Q6: Please specify the IDM (architecture, data, accuracy on real vs generated frames), report seed variance, report cache cost in GPU-hours, and provide the complete manuscript with references.

Reviewer_74ysOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper addresses uncertainty estimation for action-conditioned video world models (WMs) in robotics. The main novelty is the use of action fidelity, identified by an inverse dynamics model (IDM) that runs on the generated video and is compared against the ground-truth teleoperated actions. If the action cannot be recovered from the generated video, the uncertainty of the WM should grow.

Critical strengths

The gap the authors identify is real and important for robotics. Measuring action fidelity in addition to visual uncertainty is an important and necessary component for using WMs in planning.

Critical weaknesses
  • The IDM focuses on the robot: it retrieves what the robot did, not how the scene changed. Is this a valid signal for uncertainty? This should be clarified and properly explained. In addition, the paper does not state which IDM is used or what data it was trained on (e.g., was it trained on data independent of the evaluated subset?).
  • Figure 1: the illustrated generated video is fine - however, this is not how WMs typically fail. They usually hallucinate a successful grasp, depicting the object as grasped even when the gripper closes on air. Your illustrative example therefore does not show the problem you are solving.
  • The simulation is mentioned only very briefly, yet real-to-sim transfer is a research problem in itself. How did you do it? Also, DROID contains ~90k rollouts, but you reduced the evaluation to a few hundred. Why? Finally, how do you ensure that your simulation is a valid proxy for the real world?
  • Only teleoperated actions are used in the evaluation. They are played through the WM and through the simulation, and the simulation is taken as ground truth. Since these are actions that led to success in the real world, their replay in simulation should achieve a high success rate. It is therefore unclear why you are detecting failures of these actions.
  • The main motivation is policy evaluation or planning in the WM. Evaluation is cheap to do (it requires no real-world experiments), yet the authors omit it completely. When doing such an evaluation, you could also correlate the results against simulation benchmarks (e.g., REALM, SIMPLER, RoboLab) to further verify your model and justify the uncertainty estimates.
  • There are no qualitative results. You should qualitatively analyze the real-to-sim pipeline and show qualitative results of action fidelity on real data, etc.
Questions for authors

Please address the weaknesses.

Reviewer_xGcPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CalibWM adds a lightweight uncertainty probe to a frozen video world model. It combines inverse-dynamics-based action-fidelity supervision, spatial and temporal error estimation, and conformal calibration to detect unreliable predictions and provide prediction-error intervals.

Critical strengths
  1. The paper explicitly separates failure discrimination from error-interval coverage and evaluates their contributions through ablations.
  2. Comparisons against controls using the same features, labels, and training budget make the incremental gains easier to assess.
Critical weaknesses
  1. Modest incremental gains. Much of the improvement comes from adding action-fidelity supervision. The full method improves AUROC over the strongest same-label control by only 0.020 on Bridge and 0.017 on DROID, leaving its practical advantage relative to the added complexity unclear.

  2. Training labels rely on an inverse dynamics model, which may itself fail on generated videos. Replay-based evaluation helps, but scene reconstruction introduces additional assumptions and selection bias.

  3. Conformal coverage concerns prediction-error regions rather than correct action-failure detection. The ranking guarantee also depends on strong assumptions that are not established for the learned probe.

  4. Narrow practical validation. Experiments use a single world-model backbone and do not demonstrate improvements in downstream planning or policy evaluation. It remains unclear whether better detection scores translate into better decisions.

Questions for authors
  1. When does the full method offer a meaningful practical advantage over the simpler same-label classifier?

  2. How sensitive are the results to the inverse dynamics model and its failure threshold?

  3. Can the predicted uncertainty improve planning or policy evaluation, and does it transfer to another backbone?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.