- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces MiraBench, a benchmark for evaluating the reliability of action-conditioned robotic world models. It evaluates physical consistency, action following, and optimism bias through failure-inducing action perturbations. The authors evaluate 12 model configurations and validate their automated metrics using human annotations, showing that visual quality and task success do not necessarily imply reliable failure prediction.
Critical strengths
Evaluating whether world models correctly predict failures is valuable, as existing benchmarks mainly focus on visual quality and successful task execution. The paper evaluates 12 model configurations and includes over 16,000 human annotation judgments, providing useful insights into the limitations of current world models.
Critical weaknesses
The paper assumes that the proposed perturbations necessarily cause task failure, but this is not sufficiently validated. For example, scaling joint commands does not necessarily reduce grasp force proportionally or cause an object to slip. The metric mainly checks whether normal and perturbed predictions differ, rather than whether the perturbed prediction actually shows the expected failure. This raises concerns about whether it accurately measures optimism bias.
Questions for authors
Were the perturbed actions tested in a physics simulator to verify their actual failure rates? For cases classified as "Different" by the Bias Resistance evaluator, how often do the MB-9 human annotations confirm correct failure prediction?
3 comments
Dear Reviewer xeBJ,
Thank you again for your thoughtful review and encouraging assessment of MiraBench. Your questions helped us clarify the execution-based validation of the perturbations and the distinction between outcome sensitivity and correct failure prediction. We would welcome any further feedback on our response. If any points remain unclear or require additional explanation, please let us know. We would be happy to address them.
Best regards,
The Authors
Dear Reviewer xeBJ,
Thank you again for your thoughtful review and encouraging feedback on MiraBench. As the discussion period is coming to an end, we would like to express our appreciation for your time and helpful suggestions. If you have any further feedback on our response, we would be happy to hear it and clarify any remaining questions.
Best regards,
The Authors
Dear Reviewer xeBJ,
Thank you for the careful and constructive review. We are greatly encouraged by your recognition that evaluating failure prediction is valuable and that the model coverage and human annotation effort provide useful insights into current world models. We also sincerely appreciate your thoughtful questions about perturbation validity and the connection between outcome differences and correct failure prediction. Below, we reply to all your comments one by one.
Appendix B describes the six failure-inducing perturbation families, their task assignments, and the expected failure mode for each family. The rationale is mechanical: Deliberately large and visible action interventions, such as near-open grip, premature release, wrist-angle offset, and approach overshoot, are designed to disrupt physical preconditions for success. They are therefore interpretable intended-failure probes and a relatively lenient test of perturbation sensitivity. Here, weak grasp denotes an intervention on hand-joint commands.
To provide additional execution-based evidence, we constructed MuJoCo environments using manufacturer GR1T1/GR1T2 and Fourier-hand assets together with the official recorded joint trajectories. Across seven task categories and eleven calibrated source episodes, all 11 nominal rollouts succeeded, and all 33 perturbation rollouts failed to complete the task (100% observed failure rate under the calibrated central settings). These tests cover all six perturbation families. Task outcomes were checked from simulated object/particle states and contact traces. All 33 perturbed rollouts missed their target-region criterion. For example, in the pouring episode, the nominal rollout placed all six reconstructed particles in the basket, while each of its three strengthened perturbations placed none. Execution-based verification on physical hardware remains valuable future work. We will incorporate the validation into the camera-ready version.
The human annotations separately assess task completion, false-success prediction, and visual degradation AppendixG2˙. This audit also identifies outcome changes that still depict task success. Accordingly, a `Different` verdict measures outcome sensitivity; it does not by itself verify correct failure prediction accuracy. To examine the generation-error concern, we analyzed the existing human annotations. In the annotation, 87.7% of perturbed predictions were human-labeled false success, and automatic optimism-bias flags achieved 95.7% precision. These results support false-success detection in visually coherent predictions.
Among cases classified as Different by the Bias Resistance evaluator, 76.7% were human-labeled as showing an appropriate failure rather than a successful outcome, while most remaining cases changed outcome but were ambiguous. Thus, a Different verdict indicates outcome sensitivity but does not by itself certify correct failure prediction. This distinction is important because, across all perturbed predictions, 87.7% were human-labeled false success, showing that current models still struggle with the basic requirement of avoiding successful predictions under intended-failure interventions. Accordingly, our present focus is optimism-bias detection, i.e., whether a prediction still depicts task success under intended-failure interventions. In this direction, human task-outcome labels are sufficient to determine whether a prediction still depicts success without assigning a precise physical failure mechanism. Current models still struggle with the basic false-success problem; as they become more capable of generating coherent and diverse failure trajectories, we plan to extend MiraBench to evaluate the physical mechanism and causal fidelity of those predictions.
We sincerely appreciate your recognition and feedback. We will incorporate these results and clarify this evaluation focus in the revised paper. If these clarifications address your concerns, we would be grateful for your reconsideration. We look forward to your support in advancing this work within the research community!