- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work introduces CalibWM, a lightweight probe attached to a frozen action-conditioned video world model to produce spatiotemporal uncertainty maps across latent units and prediction horizons. The probe is optimized using a suite of carefully designed loss functions, including horizon-aware regularization, spatial smoothing losses, proper scoring and distillation losses leveraging cached teacher outputs, and an alignment loss for learning a scalar readout that predicts action-fidelity failures, where such failures are identified by whether a fixed inverse dynamics model cannot recover executed actions from synthetic frames. The method incorporates a segment-level split-conformal mechanism to establish finite-sample marginal coverage guarantees for its latent prediction-error regions, and it provides a further theoretical result bounding the readout’s discriminative performance under stated consistency and proxy assumptions. Empirically, CalibWM achieves strong discrimination performance against replay labels, outperforming both the strongest label-free baseline and the best label-using control; it also yields favorable results on a human-annotated optimism-bias subset, maintains near-target segment-level coverage, improves horizon-dependent uncertainty behavior, and supports out-of-distribution detection via its epistemic uncertainty branch.
Critical strengths
The paper has several strengths: it addresses a well-motivated problem, arguing that uncertainty for world models should reflect action outcomes rather than only visual error; it attempts to separate label access from design by introducing two matched label-using controls together with trajectory-clustered conformal intervals; it features rigorous and carefully scoped conformal treatment, where Proposition 1 is correct and the work clearly distinguishes guaranteed from non-guaranteed claims as well as marginal slack from Beta fluctuation; it pre-specifies falsification criteria (Table 3), and horizon regularization delivers a clear width benefit (ρ 0.96 vs 0.61; h=16 width ratio 1.21 vs 2.43); additionally, the method is cheap at inference, requiring only a single forward pass and fewer than 1% of backbone parameters.
Critical weaknesses
-
Label validity is the key weakness. The primary evaluation labels (simulator replay of recorded actions) never use the generated video. Table 1 may measure sim-replay difficulty rather than world-model action fidelity. κ between weak and replay labels is never reported.
-
It is unclear how C³-backbone features are obtained for videos from other generators. There is possible τ_a leakage, no CIs, and the visual-quality selection makes the LPIPS comparison weak by construction.
-
Small design-attributable gain. It is +0.02 over the best label-using control, confounded by object masks and extra ground-truth supervision. Without the label, CalibWM is no better than C³.
-
Proposition 2 is near-tautological. Assumption 2 presumes Bayes-optimal ranking through a scalar bottleneck. The proposition is stated for the IDM label but evaluated on replay labels, and its falsification test is effectively not at risk.
-
Coverage result has no demonstrated use. It guarantees only a latent-space tolerance, does not affect the readout, and has no downstream use. Baselines are not conformalized for a matched-width comparison.
-
Inconsistencies and gaps. Width 0.82 vs 0.88; π changes despite a post-hoc layer; a 14-point coverage drop despite ρ = 0.96; possible overlap between training and calibration splits. A single backbone, an unspecified IDM, no seed variance, a substantial training-time cache cost, and an incomplete manuscript.
Questions for authors
Q1: Does the replay label use the generated rollout at all? What fraction of segments would be labelled failures for a perfect world model? Please report Cohen's κ and an action/context-only "difficulty" baseline.
Q2: How are C³ penultimate features and action embeddings computed for MiraBench videos from other generators? Which segments were used to fit τ_a? Please give positive counts and clustered CIs.
Q3: Do the label-using controls use the same object-mask aggregator, ranking surrogate, weighting, and schedule? Where do masks come from at deployment?
Q4: Which Y does Proposition 2 refer to? Are masks part of Φ? Would you test Assumption 2 against a high-capacity classifier on full Φ? Are probe-training and calibration segments disjoint (§4.1 suggests both use the full cache)?
Q5: Please explain the 0.82/0.88 width discrepancy, the π change under a post-hoc layer, and the coverage collapse of the w/o-conformal variant despite ρ = 0.96. Please define ECE.
Q6: Please specify the IDM (architecture, data, accuracy on real vs generated frames), report seed variance, report cache cost in GPU-hours, and provide the complete manuscript with references.