- Rating
- 3
- Confidence
- 1(−1)
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Action-conditioned video world models coupled with vision-language model (VLM) judges are increasingly used to evaluate robot policies without physical trials. While prior benchmarks validate these evaluators via high correlation with real success rates, the paper investigates whether statistical correlation genuinely certifies sensitivity to action consequences.
Empirical results show that auditing a state-of-the-art evaluator (Wan2.1 video model + Qwen2.5-VL judge on LIBERO) across 31 policies reveals critical failure modes:
(1) Intervention Insensitivity: Despite showing correlation across 24 chunked action policies, the evaluator altered its judgment in only of cases under interventions designed to cause guaranteed physical task failure.
(2) Perturbation Blindness: A test-time grasp-delay perturbation dropped simulator success from to while counterintuitively raising the evaluator score from to .
(3) Demonstration Bias: Evaluator scores correlate strongly () with how closely generated rollouts resemble the training demonstrations, rewarding memorization rather than physical correctness.
The authors prove that without additional environment interactions, no evaluator can reliably rank two candidate policies whose distinction depends on actions unsupported by demonstrations. This fundamental unidentifiability holds even assuming candidate weights, an optimal pretrained world model, and an error-free judge. Controlled experiments verify that scaling demonstrations without coverage of the critical action fails to decrease error.
Critical strengths
(1) The phenomenon identified in the paper is interesting and worth exploring, which will inspire new research effort on rethinking the role of action-conditioned generative model in robotics.
(2) The experiments are extensive and findings in the experiments including Perturbation Blindness and Demonstration Bias demonstrate the meaningfulness of this problem.
(3) The authors provide theoretical analysis on the findings.
Critical weaknesses
(1) Most of the experiments focus on simulation environment. Results can be more supportive if there is real-world experiments.
(2) The proposed ResGap can only serve as coarse-screening. Is it possible for us to find a space that can well describe the relationship between the rollouts of generative model and success rate in target environment? Or does it stem from the ability of current generative model?
Questions for authors
Please refer to the critical weakness part.
2 comments
Reviewer heR3 please respond to author comments
Reviewer heR3 please respond to author comments
Thank you for reviewing our paper. We agree that real-world experiments would further strengthen our findings, and we consider them a top priority for future work. Although our experiments are conducted entirely in simulation, they demonstrate that correlation with success rate does not guarantee sensitivity to action consequences, consistent with our theoretical results.
Your question about which space could capture the relationship between generative-model rollouts and success rates in the target environment is precisely what we are currently investigating. Our results show that even when generated videos resemble successful demonstrations, they do not necessarily reflect the consequences of the given actions accurately. This motivates us to investigate how the relationship between observations and actions is reflected in policy parameter adaptation, in addition to video similarity. Inspired by ProjNorm and MAML, we believe that capturing the relationship between parameter changes and success rates in the target environment could lead to a useful evaluation metric. ResGap is a first step toward exploring this possibility.
Improvements in generative models may also mitigate the failure patterns observed in this study. However, parameter-based evaluation is also subject to the limitations of Theorem 1 when the available information is the same. Clarifying the conditions under which such metrics can predict success rates therefore remains an important direction for future work.