- Rating
- 2
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper introduces VOE-Bench, an evaluation framework for auditing whether conformally calibrated latent world model prediction errors can differentiate physical inconsistencies from benign changes. The authors experimented across 3 environments, and show that sensitivity to calibration references, appearance-dependent detection, and find no significant improvement in downstream planning performance.
Critical strengths
-
- The paper performs a rigorous set of experimental controls (nominal calibration, physical selectivity, decision utility), strengthening the credibility of empirical findings.
-
- The paper provides strong evidence of calibration sensitivity and object dependent effects, with quantitative analyses and uncertainty estimates. These findings can be useful to understand limitations of interpreting latent prediction errors as indicators of physical violations.
Critical weaknesses
-
- Despite being among the central contributions, technical details and justification for VOE bench is severely limited in the main text (most are buried deep in the appendix). Furthermore, the environments and several interventions have been explored in prior works, making it challenging the contribution and novelty of the benchmark and its analyses.
-
- While the authors audit 29 checkpoints, key findings rely on a small number of selected models and environments. The proposed diagnostics do not translate into validated online monitoring methods, and downstream planning experiments demonstrate no statistically significant improvements.
Questions for authors
-1. Could the authors clarify which aspects of the benchmarks and its empirical findings constitute genuinely new contributions beyond existing violation-of-expectation evaluations? A more explicit comparison would help assess the significance and novelty of the contribution.
- 2. To what extent do the reported phenomena generalize across the other model families and environments? This is important to assess whether the paper establishes general limitations of latent surprise.
-
- The experiments primarily use the upstream PushT checkpoint, which exhibits relatively weak action-mismatch detection. Could the authors clarify how this choice affects the interpretation of the negative planning results.
-
- The paper defines timely detection using the interval [j, j+2], while acknowledging that an action mismatch cannot affect the observed position until j+1. Could the authors clarify how many reported timely action-mismatch alarms occur at j, before the intervention becomes observable? Would excluding such alarms materially change the primary predictor comparison?
-
- IN Section 2.2, it appears that PushT contributed windows to world-model training, so does this mean that the evaluation is not independent of model training? If so, this is a potential confound.
-
- The description for VOE bench and its itergration needs to be incorporated to the main paper, with much clearer explanation for readers to digest.