- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents JEPA-UQ, a measurement protocol designed to assess uncertainty in latent world-model rollouts. The protocol asks that uncertainty claims include four distinct components and separates such claims into three tiers. The empirical study evaluates Level A directly, investigates Level B, and treats Level C as an extension to the protocol. The protocol is well considered and the experiment is rigorously checked, but the empirical coverage is still too limited to support benchmark-level claims. Expanding to more encoders and datasets, adding a fixed-predictor track, including more rare events, and performing downstream validation would significantly strengthen the submission. The paper is likely to be at the borderline, I lean towards "weak rejection" but would be happy with "weak acceptance" as well if other reviewers suggest so.
Critical strengths
-
The paper presents a well-defined framework that differentiates between ordinal risk scores, calibrated probabilities, and joint predictive distributions.
-
Its clear distinction between training, development, calibration, and evaluation datasets is outstanding. It also appropriately manages overlapping windows through scene-level clustered resampling.
-
It clearly discloses horizon-specific findings, counts of rare events, limits on uncertainty, and any corrections made after the freeze. The conclusions are suitably cautious: a continuous uncertainty ranking is still informative, but failure detection is mainly supported at shorter horizons.
Critical weaknesses
-
It assesses just one encoder, a single predictor family, one dataset, one camera perspective, and two closely related variance-based uncertainty approaches.
-
Because the ensemble and MC-dropout methods produce different predictions and failure labels, the study cannot disentangle the quality of the uncertainty estimator itself.
-
Evaluation at longer time horizons is statistically underpowered, with just 6–20 failure events. Level B lacks strong validation, and Level C has not been deployed. Latent MSE is not tied to any physical or decision-relevant downstream endpoint.
Questions for authors
Do the findings remain consistent across different calibration quantiles?
In what way does latent MSE correspond to physical or semantic prediction error?