acceptodds
Under review as a conference paper at ICLR 2027

MiraBench: Evaluating Action-Conditioned Reliability in Robotic World Models

Abstract

Action-conditioned world models are increasingly used as scalable simulators for robot learning, yet current evaluations provide limited evidence that their predictions are reliable under the actions they condition on. Existing benchmarks largely emphasize visual fidelity, leaving unclear whether predicted futures are physically plausible, faithful to commanded actions, and calibrated to failure when actions should not succeed. We introduce MiraBench, a benchmark that defines action-conditioned reliability as a core evaluation target for robotic world models. MiraBench evaluates three complementary dimensions: Physics Adherence, which evaluates reference-free physical consistency; Action-Following Fidelity, which measures whether predictions respect task-relevant action inputs; and Optimism Bias Detection, which probes the tendency to predict successful outcomes under failure-inducing actions. To support this evaluation, we curate a human-annotated corpus with over 16,000 judgments across tasks, failure categories, and leading world models. We evaluate 12 representative model configurations spanning vector-conditioned robotic world models, text-conditioned generative world models, open-weight systems, closed-source systems, and multiple model scales. Across this broad model landscape, MiraBench reveals three central findings: visual fidelity is a poor proxy for action fidelity; increasing model scale does not reliably improve action following; and optimism bias is pervasive across current systems. By shifting evaluation from appearance to action-conditioned reliability, MiraBench provides a diagnostic foundation for assessing and improving robotic world models as faithful simulators.

Primary area:
datasets and benchmarks
Keywords:
World Models, Action-Following, Generative Models, Optimism Bias

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_xeBJOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces MiraBench, a benchmark for evaluating the reliability of action-conditioned robotic world models. It evaluates physical consistency, action following, and optimism bias through failure-inducing action perturbations. The authors evaluate 12 model configurations and validate their automated metrics using human annotations, showing that visual quality and task success do not necessarily imply reliable failure prediction.

Critical strengths

Evaluating whether world models correctly predict failures is valuable, as existing benchmarks mainly focus on visual quality and successful task execution. The paper evaluates 12 model configurations and includes over 16,000 human annotation judgments, providing useful insights into the limitations of current world models.

Critical weaknesses

The paper assumes that the proposed perturbations necessarily cause task failure, but this is not sufficiently validated. For example, scaling joint commands does not necessarily reduce grasp force proportionally or cause an object to slip. The metric mainly checks whether normal and perturbed predictions differ, rather than whether the perturbed prediction actually shows the expected failure. This raises concerns about whether it accurately measures optimism bias.

Questions for authors

Were the perturbed actions tested in a physics simulator to verify their actual failure rates? For cases classified as "Different" by the Bias Resistance evaluator, how often do the MB-9 human annotations confirm correct failure prediction?

3 comments

Dear Reviewer xeBJ,

Thank you for the careful and constructive review. We are greatly encouraged by your recognition that evaluating failure prediction is valuable and that the model coverage and human annotation effort provide useful insights into current world models. We also sincerely appreciate your thoughtful questions about perturbation validity and the connection between outcome differences and correct failure prediction. Below, we reply to all your comments one by one.


W1 The assumption that the proposed perturbations necessarily cause task failure is not sufficiently validated.

Q1(a) Were the perturbed actions tested in a physics simulator to verify their actual failure rates?

Appendix B describes the six failure-inducing perturbation families, their task assignments, and the expected failure mode for each family. The rationale is mechanical: Deliberately large and visible action interventions, such as near-open grip, premature release, wrist-angle offset, and approach overshoot, are designed to disrupt physical preconditions for success. They are therefore interpretable intended-failure probes and a relatively lenient test of perturbation sensitivity. Here, weak grasp denotes an intervention on hand-joint commands.

To provide additional execution-based evidence, we constructed MuJoCo environments using manufacturer GR1T1/GR1T2 and Fourier-hand assets together with the official recorded joint trajectories. Across seven task categories and eleven calibrated source episodes, all 11 nominal rollouts succeeded, and all 33 perturbation rollouts failed to complete the task (100% observed failure rate under the calibrated central settings). These tests cover all six perturbation families. Task outcomes were checked from simulated object/particle states and contact traces. All 33 perturbed rollouts missed their target-region criterion. For example, in the pouring episode, the nominal rollout placed all six reconstructed particles in the basket, while each of its three strengthened perturbations placed none. Execution-based verification on physical hardware remains valuable future work. We will incorporate the validation into the camera-ready version.


W2 The metric checks whether normal and perturbed predictions differ, rather than whether the perturbed prediction shows the expected failure.

Q1(b) For cases classified as Different by the Bias Resistance evaluator, how often do the MB-9 human annotations confirm correct failure prediction?

The human annotations separately assess task completion, false-success prediction, and visual degradation AppendixG2˙Appendix G\.2. This audit also identifies outcome changes that still depict task success. Accordingly, a `Different` verdict measures outcome sensitivity; it does not by itself verify correct failure prediction accuracy. To examine the generation-error concern, we analyzed the existing human annotations. In the annotation, 87.7% of perturbed predictions were human-labeled false success, and automatic optimism-bias flags achieved 95.7% precision. These results support false-success detection in visually coherent predictions.

Among cases classified as Different by the Bias Resistance evaluator, 76.7% were human-labeled as showing an appropriate failure rather than a successful outcome, while most remaining cases changed outcome but were ambiguous. Thus, a Different verdict indicates outcome sensitivity but does not by itself certify correct failure prediction. This distinction is important because, across all perturbed predictions, 87.7% were human-labeled false success, showing that current models still struggle with the basic requirement of avoiding successful predictions under intended-failure interventions. Accordingly, our present focus is optimism-bias detection, i.e., whether a prediction still depicts task success under intended-failure interventions. In this direction, human task-outcome labels are sufficient to determine whether a prediction still depicts success without assigning a precise physical failure mechanism. Current models still struggle with the basic false-success problem; as they become more capable of generating coherent and diverse failure trajectories, we plan to extend MiraBench to evaluate the physical mechanism and causal fidelity of those predictions.


We sincerely appreciate your recognition and feedback. We will incorporate these results and clarify this evaluation focus in the revised paper. If these clarifications address your concerns, we would be grateful for your reconsideration. We look forward to your support in advancing this work within the research community!

Dear Reviewer xeBJ,

Thank you again for your thoughtful review and encouraging assessment of MiraBench. Your questions helped us clarify the execution-based validation of the perturbations and the distinction between outcome sensitivity and correct failure prediction. We would welcome any further feedback on our response. If any points remain unclear or require additional explanation, please let us know. We would be happy to address them.

Best regards,
The Authors

Dear Reviewer xeBJ,

Thank you again for your thoughtful review and encouraging feedback on MiraBench. As the discussion period is coming to an end, we would like to express our appreciation for your time and helpful suggestions. If you have any further feedback on our response, we would be happy to hear it and clarify any remaining questions.

Best regards,

The Authors

Reviewer_asy3OpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that action-conditioned world models are evaluated mostly on visual realism, and proposes action-conditioned reliability as the target instead. It splits that target into three parts: Physics Adherence, Action-Following Fidelity, and Optimism Bias Detection. The authors curate 906 generated videos and 16,704 human annotation decisions over GR-1 humanoid episodes, validate zero-shot VLM judges against those labels, and score 12 model configurations. The reported findings are that visual fidelity is a poor proxy for action fidelity.

Critical strengths
  1. The evaluation target is the right one. Asking whether a world model predicts failure when the action implies failure is different from asking whether the video looks good, and it is the question that matters if these models are to be used as simulators.

  2. The corpus and the three-way split are a usable asset. 906 videos with 16,704 structured decisions is enough to calibrate judges, and separating physics, action-following, and failure sensitivity stops the reader from collapsing three different failure modes into one score.

Critical weaknesses
  1. The ground truth for failure is asserted, not observed. Appendix B.4 states that each perturbation "necessarily prevents task completion" under correct physics. No perturbed action is ever replayed in Isaac Lab or on hardware to check that claim. The operationalised judgment is also weaker than Eq. 3. The judge is asked whether the object ends up in the same general location, not whether the task succeeded, so a different but still successful outcome counts as failure-preserving.

  2. The headline metric is ungated, and the paper admits this only in the appendix. Appendix L.1 says the Table 2 score is ungated and that a model can separate nominal from perturbed rollouts without completing the nominal task. DreamDojo-2B has task completion 14.0 and Bias Resistance 48.7; that number is largely failure by incapacity. Gated values exist in Appendix L but never reach Table 2. The scale result (48.7 → 12.8) therefore has a trivial alternative explanation: the larger model renders success more often, so it looks more optimistic.

Questions for authors

Q1. What fraction of "Different" verdicts do you get (a) between two nominal rollouts of the same episode with different seeds, and (b) between a nominal rollout and a benign action edit that still succeeds? Without those two base rates, how do you separate failure insensitivity from general action insensitivity?

Q2. Have you replayed any of the six perturbations in simulation or on hardware to confirm they fail?

Q3. If the text-conditioned systems are re-scored with the standard prompt and a style-matched reference for object preservation, do Findings 1 and 4 survive?

Q4. Which corpus produced the 906 videos? Appendix A describes 50 microwave-door demonstrations, Section 5.1 lists nine pick-and-place and folding tasks, and Appendix I mentions a 22,209-episode teleoperation corpus.

8 comments

Dear Reviewer asy3,

Thank you for the careful and detailed review. We are greatly encouraged by your recognition that action-conditioned reliability is the right evaluation target, and that the corpus and the three-way diagnostic design are useful assets. Below, we reply to all your concrete comments one by one.


W1(a) The ground truth for failure is asserted rather than observed, and the operational judgment is weaker than Eq. 33.

Q2 Have you replayed any of the six perturbations in simulation or on hardware to confirm they fail?

Thank you for this helpful suggestion. We have added a MuJoCo execution study covering seven task categories and eleven episodes, with scenes calibrated using nominal trajectories. All 11 nominal rollouts succeeded, and all 33 perturbation rollouts failed (100% observed failure rate). These results provide execution-based validation of the failure-guarantee in the tested reconstructed scenes. Hardware verification remains valuable future work.

To connect the automatic diagnostic with the task-outcome definition in Eq. 33, we complement nominal/perturbed outcome comparisons with human assessments of task completion and visual plausibility. According to the annotation, 87.7% of perturbed predictions were labeled as false success by human annotators. Automatic optimism-bias flags achieved 95.7% precision. These results demonstrate that our judge identifies false-success predictions with high precision.

W1(b) The judge is asked whether the object ends up in the same general location, not whether the task succeeded, so a different but still successful outcome counts as failure-preserving.

Thank you for highlighting this distinction. An important feature of our evaluated tasks is that successful outcomes are relatively concentrated and visually distinguishable. The tasks specify concrete goals, such as placing an object in a designated container or target region, which constrain success to a narrow set of recognizable end states rather than many qualitatively different outcomes. Successful rollouts may vary in trajectory, exact pose, or position within that region, while satisfying the same goal-defined outcome. The evaluator explicitly treats the same container or area as `Same`, so these ordinary variations within a successful outcome do not themselves constitute a `Different` verdict. It compares task-relevant states of the manipulated object over seven late-phase comparisons, ignoring blur, color, and rendering differences AppendixD3˙Appendix D\.3. This design focuses the comparison on meaningful changes in the object's task-relevant state rather than exact visual matching.

Human evaluation then checks task success directly, complementing the object-state comparison. Human annotations separately assess nominal and perturbed task completion, false-success prediction, and visual degradation AppendixG2˙Appendix G\.2. This allows the human audit to identify changed outcomes that still depict task success, including cases receiving an automatic `Different` verdict. We report automatic outcome sensitivity and human-confirmed task outcomes separately. According to human validation, 87.7% of perturbed predictions were human-labeled false success (the rest are mostly ambiguous ones), and automatic optimism-bias flags achieved 95.7% precision. These results support the evaluator's ability to detect false success.


W2 The headline metric is ungated, and the paper admits this only in the appendix. Appendix L.1 says the Table 2 score is ungated and that a model can separate nominal from perturbed rollouts without completing the nominal task. DreamDojo-2B has task completion 14.0 and Bias Resistance 48.7; that number is largely failure by incapacity. Gated values exist in Appendix L but never reach Table 2. The scale result 487˙→128˙48\.7 → 12\.8 therefore has a trivial alternative explanation: the larger model renders success more often, so it looks more optimistic.

Thank you for raising this important concern. We agree that nominal task competence can confound the interpretation of ungated Bias Resistance, particularly for smaller models. MiraBench separates nominal task completion L2L2 from sensitivity to failure-inducing action perturbations L3L3. The ungated L3 metric evaluates this sensitivity over the full test set, while the conditional metric isolates cases where the model successfully completes the nominal task. We acknowledge that DreamDojo-2B's 48.7% Bias Resistance, alongside its 14.0% TCR, should be interpreted with this distinction in mind. The ungated comparison alone cannot establish that scaling causes a decline in perturbation sensitivity.

Importantly, our conditional analysis in Appendix L.1 and Table 23 provides direct evidence that optimism bias persists even when nominal task generation succeeds. Specifically, we retain only nominal baselines satisfying TCR = 1 and Obj. ≥ 30, and evaluate their three paired perturbations using the original L3 criterion. Under this stricter setting, DreamDojo-GR1-14B achieves only 14.8% conditional Bias Resistance across 27 perturbations associated with nine successful nominal baselines. Since every retained baseline successfully completes its task, this result demonstrates that the model can generate the intended nominal outcome yet still fail to respond appropriately to failure-inducing interventions. Thus, nominal generation incapacity cannot fully explain the observed optimism bias.

The conditional results also reveal meaningful differences across models. DreamDojo-GR1 changes from 18.2% to 14.8%, whereas Cosmos-GR1 increases from 22.2% to 44.4%. These results suggest that nominal competence and perturbation sensitivity are distinct capabilities whose relationship varies across models. We agree that the small conditional sample sizes warrant caution, and we will avoid attributing the observed trends to model scale alone.

In the revision, we will move conditional Bias Resistance and its sample counts into Table 2, alongside the ungated results, and explicitly clarify their interpretations. We will also refine the scaling discussion to emphasize the conclusion directly supported by our evidence: Improved nominal task competence does not necessarily translate into greater sensitivity to failure-inducing action changes. This distinction is central to MiraBench's motivation for evaluating these capabilities separately.


Q1 What fraction of Different verdicts occurs between nominal rollouts with different seeds, and between nominal and benign edits? How is failure insensitivity separated from general action insensitivity?

Thank you for this insightful suggestion. The seed controls indicate limited sensitivity to generation randomness. The assessable small-edit and nominal-repeat controls yield 0.0% `Different` verdicts. The results are as follows:

ModelDifferent-seed nominal: Different rateSmall-edit / nominal-repeat: Different rate
DreamDojo-GR1-2B7.7%0.0%
DreamDojo-GR1-14B0.0%0.0%
Cosmos-GR1-2B15.4%0.0%
Cosmos-GR1-14B0.0%0.0%
Cosmos-GR1-14B-I2V0.0%0.0%
Wan2.1-I2V-14B0.0%0.0%
Wan2.2-TI2V-5B7.7%0.0%

For how failure insensitivity is separated from general action insensitivity, our present focus is optimism-bias detection, i.e., whether a prediction still depicts task success under intended-failure interventions. L2 separately measures nominal action following, while the controls above quantify sampling variability and responses to small input changes. Human task-outcome annotations then check false-success predictions. We therefore interpret L3 jointly with L2 and these controls: `Different` denotes outcome sensitivity, while false-success labels identify optimism bias. Current models still struggle with this basic false-success problem. As they become more capable of generating coherent and diverse failure trajectories, we plan to extend MiraBench to evaluate the physical mechanisms and causal fidelity of those predictions.

Q3 If text-conditioned systems are re-scored with the Standard Prompt and a style-matched reference for object preservation, do Findings 1 and 4 survive?

Thank you for this constructive suggestion. We will clarify two points.

First, we have carefully validated our VLM judges by human annotation. On 906 generated videos and 16,704 structured human judgments, the per-metric VLM–human agreement is between 85.1–92.3% optimismbias:878˙optimism bias: 87\.8% on held-out expert labels. Moreover, the lenient protocol for text-conditioned models was validated separately. Details are already provided in Appendix D.3 (Tables 10–12) and Appendix G.5.

Second, regarding style-matched references for OPS, we agree that style mismatch is a legitimate concern, but the reference should still be the original GT video. GT is model-independent ground truth. In comparison, a style-matched reference would inherit generator or rendering priors, may share failure modes with the evaluated model, and could introduce color, texture, and lighting variation unrelated to object preservation, creating a form of circular validation. Therefore we keep the original GT as the primary protocol and handle cross-style differences through the validated lenient judge prompt.


Q4 Which corpus produced the 906 videos, given the microwave-door demonstrations, nine manipulation tasks, and 22,209-episode teleoperation corpus?

Thank you for the question. The source episodes come from the GR-1 humanoid corpus and Lingchu bimanual dataset Section 5\.1, p\. 6. Their initial observations and action conditions are used to generate world-model rollouts, which are then human-annotated. The reported total of 906 refers to these generated evaluation videos across models and annotation modules, as documented in Appendix G.5 p\. 39.

The three other figures describe different parts of the pipeline. The 50 microwave-door demonstrations are the size of the Arena-GR1 dataset described in Appendix A.1 p\. 16. The nine tasks specify the L3 failure-perturbation schedule Appendix B\.2, Table 5, p\. 18, rather than the task coverage of all four annotation modules. The 22,209 teleoperation episodes are the PhysicalAI-GR1 data used for Cosmos post-training Appendix I\.1, “Training data,” p\. 44.

The 906 generated videos were curated for evaluator validation, pooling outputs across different models and tasks to cover diverse generation behaviors. The human-grounded validation therefore spans vector-conditioned DreamDojo-2B/14B, text-conditioned open-weight Wan2.1 and closed-source Happy Horse, and a DreamDojo-14B/Lingchu subset for task-domain transfer. VLM–human agreement is 87.3%–96.2% across the four main validation groups Appendix D\.3, Table 12, p\. 30, providing validation across conditioning interfaces, model scales, access types, and task domains.


We sincerely appreciate your careful review and recognition of MiraBench's evaluation target and human-annotated corpus. We will incorporate the execution evidence, conditional analysis, and data-source clarifications into the main presentation.

We hope these clarifications and additional evidence help address your concerns, and would be grateful for your reconsideration. We look forward to your support in advancing this work within the research community!

Reviewer_E9a6OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MiraBench evaluates robotic world models through physical consistency, action following, and failure prediction. It tests 12 model configurations using nominal and perturbed actions, with VLM judges and kinematic checks supported by human annotations.

Critical strengths
  1. Evaluating failure-inducing actions is a useful addition to benchmarks focused on visual quality and successful task execution.
  2. The multiple diagnostic dimensions, model coverage, and structured human annotations make the benchmark a useful resource.
Critical weaknesses
  1. The failure assumptions need stronger validation. It is unclear whether every perturbation necessarily causes failure; for example, scaling joint commands does not directly establish reduced grip force. Execution-based verification would strengthen these assumptions.

  2. Outcome changes do not necessarily indicate correct failure prediction. The optimism-bias evaluator mainly compares object states between nominal and perturbed videos. Unrelated changes or generation errors could therefore receive favorable scores without depicting the expected failure.

Questions for authors
  1. Were the perturbed actions executed to verify that they produce the intended failures?
  2. How does the evaluator distinguish the expected failure from unrelated changes or generation errors?
4 comments

Dear Reviewer E9a6,

Thank you for the careful and constructive review. We are greatly encouraged by your recognition that failure-inducing actions substantially complement existing evaluation focused on visual quality and successful execution, and that the diagnostic dimensions and human annotations make MiraBench a useful resource. Below, we reply to all your concrete comments one by one.


W1 The failure assumptions need stronger validation. It is unclear whether every perturbation necessarily causes failure; for example, scaling joint commands does not directly establish reduced grip force. Execution-based verification would strengthen these assumptions.

Q1 Were the perturbed actions executed to verify that they produce the intended failures?

Appendix B describes the perturbation design, including the six failure-inducing perturbation families, their task assignments, and the expected failure mode for each family. The rationale is mechanical rather than post-hoc: These are deliberately large and visible action interventions, such as near-open grip, premature release, wrist-angle offset, and approach overshoot, designed to remove physical preconditions for success.

To provide additional execution-based evidence, we constructed MuJoCo environments using manufacturer GR1T1/GR1T2 and Fourier-hand assets together with the official recorded joint trajectories. Across seven task categories and eleven calibrated source episodes, all 11 nominal rollouts succeeded, and all 33 perturbation rollouts failed to complete the task 100100% observed failure rate at the nominal-calibrated central settings. These tests cover all six perturbation families. Task outcomes are checked from simulated object/particle states and contact traces and verified by human. All 33 perturbed rollouts missed their target-region criterion. For example, in the pouring episode, the nominal rollout placed all six reconstructed particles in the basket. Execution-based verification on physical hardware remains valuable future work. We will incorporate the validation into the camera-ready version.


W2 Outcome changes do not necessarily indicate correct failure prediction. The optimism-bias evaluator mainly compares object states between nominal and perturbed videos. Unrelated changes or generation errors could therefore receive favorable scores without depicting the expected failure.

Q2 How does the evaluator distinguish the expected failure from unrelated changes or generation errors?

Thank you for highlighting this distinction. The automatic evaluator focuses on task-relevant states of the manipulated object, ignoring blur, color, and rendering differences AppendixD3˙Appendix D\.3. Human annotations separately assess nominal and perturbed task completion, false-success prediction, and perturbed realism and visual degradation AppendixG2˙Appendix G\.2. To examine the generation-error concern, we analyzed the existing human annotations. According to human annotation, it turned out that 87.7% of perturbed predictions were human-labeled false success, and automatic optimism-bias flags achieved 95.7% precision. These results support false-success detection of the evaluator.

Our present focus is optimism-bias detection, i.e., whether a prediction still depicts task success under intended-failure interventions. Nominal action-following ability is evaluated separately in L2, and the L3 diagnostic should be interpreted together with those results. We interpret `Different` as outcome sensitivity, rather than confirmation that the intended physical failure has been faithfully reproduced. Current models still struggle with the basic false-success problem; as they become more capable of generating coherent and diverse failure trajectories, we plan to extend MiraBench to evaluate the physical mechanism and causal fidelity of those predictions.


We sincerely appreciate your recognition and feedback. We will incorporate these results and clarify the relationship between perturbation sensitivity and human task-outcome validation in the camera-ready version. If these clarifications address your concerns, we would be grateful for your reconsideration. We look forward to your support in advancing this work within the research community!

Dear Reviewer E9a6,

Thank you again for your thoughtful review and encouraging assessment of MiraBench. We appreciate your constructive suggestions on perturbation validity and failure evaluation. We have provided additional execution-based evidence and human-annotation analyses in our response. Please let us know if any questions or concerns remain, and we would be happy to clarify them during the discussion.

Best regards,
The Authors

Dear Reviewer E9a6,

Thank you again for your thoughtful review and encouraging assessment of MiraBench. As the discussion period is coming to an end, we would welcome any further thoughts on our responses and additional analyses, if you have a chance. Please let us know if anything remains unclear. We would be happy to provide further clarification. We sincerely appreciate your time and constructive suggestions.

Best regards,

The Authors

Thank the authors for the answers; they clarified most of my questions. I would keep my score.

Comments on the paper

2 comments
AuthorsOpenReview ↗

A heartfelt general comment from the authors

Dear Reviewers and Area Chairs,

We sincerely thank Reviewers E9a6, asy3, and xeBJ for their thoughtful reviews and constructive suggestions. We are encouraged by the recognition that evaluating action-conditioned reliability and failure prediction addresses an important need beyond visual-quality evaluation, and that MiraBench’s diagnostic dimensions and human-annotated corpus provide a useful resource. We have taken the reviewers’ questions seriously and devoted substantial effort to providing additional experimental evidence.

During the rebuttal, we constructed nominal-calibrated MuJoCo scenes spanning seven task categories and all six perturbation families. All 11 nominal rollouts succeeded, while all 33 perturbed rollouts failed at the calibrated central settings. We also analyzed the human annotations: 87.7% of audited perturbed predictions were labeled false success, automatic optimism-bias flags achieved 95.7% precision. To address competence confounds and sampling variability, we reported conditional Bias Resistance together with nominal-repeat, small-edit, and different-seed controls. We further clarified the judge protocols and the provenance of the 906 generated videos and 16,704 structured human judgments.

We have worked through the concerns one by one, using additional executions, control results, human-outcome analyses, and protocol explanations to make the paper stronger and easier to assess. We sincerely hope that these efforts and the resulting evidence will receive consideration. We remain grateful for the reviewers’ suggestions and would warmly welcome any additional discussion.

Thank you again for your time and for helping us improve MiraBench.

Best regards,

The Authors

AuthorsOpenReview ↗

Author Final Remarks

Dear Reviewers, Area Chairs, Senior Area Chairs, and Program Chairs,

We sincerely appreciate the time and effort devoted to reviewing MiraBench. The feedback prompted us to add simulation-based execution evidence, human analyses of false-success and Different predictions, competence-conditioned results, and controls for sampling variability and small input changes. We also clarified the evaluator protocols and data provenance, providing substantive responses to the questions raised throughout the reviews.

MiraBench advances evaluation from visual appearance toward action-conditioned reliability. Through three complementary diagnostic dimensions, twelve model configurations, and 16,704 structured human judgments, it makes failures of action following and false-success prediction visible and measurable. The additional rebuttal experiments strengthen this contribution by connecting the intended perturbations with executed outcomes and separating nominal competence from responses to failure-inducing actions.

We have made every effort to strengthen the work through the review process and will incorporate the additional evidence and clarifications into the revised presentation. We remain sincerely open to further discussion and would be grateful for your recognition and support of MiraBench’s contribution to evaluating robotic world models.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.