acceptodds
Withdrawn from ICLR 2027

WorldOracle: Benchmarking Action-Conditioned World Models against Real-World Executions

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

We introduce WorldOracle, a benchmark specifically designed to evaluate action-conditioned world models (ACWMs) against paired real-world executions. Given the same initial observation and low-level action sequence, WorldOracle asks whether a predicted rollout reproduces what happens in the real world, including physical consequences and task-state transitions. We evaluate models along four complementary dimensions: physical state correctness, interaction spatial alignment, key interaction area fidelity, and multi-view action coherence. To support this evaluation, we curate real-world training and evaluation suites that cover different interaction dynamics and embodiments. The evaluation suite contains long, multi-segment episodes with both successful and failed executions, enabling evaluation across varied action outcomes and behavior distributions. Experiments on seven representative ACWMs reveal distinct model strengths and failure modes. WorldOracle provides a systematic foundation for measuring and advancing action-conditioned world modeling for manipulation. We will release the datasets, evaluation code, and leaderboard.

Primary area:
datasets and benchmarks
Keywords:
Benchmark, World Models, Robotics

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_pyN4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

WorldOracle evaluates action-conditioned world models against recorded real-world executions, using the same initial observation and low-level action sequence. The paper introduces an AgiBot G2 training set and 335 evaluation episodes spanning three embodiments and four tasks across the suite, with human-operated and policy-generated actions. Seven models are evaluated using four metrics for physical outcomes, spatial alignment, interaction-region fidelity, and multi-view alignment against real references. The results show performance degradation over long rollouts and a gap between following robot motion and predicting object interactions.

Critical strengths
  1. Pairing recorded low-level actions with real executions gives the benchmark a clear purpose. The failed attempts are particularly useful: they expose cases where a model predicts a successful interaction despite actions that failed in reality.

  2. The suite covers several interaction types, three embodiments, long human-operated episodes, and a separate policy-rollout split. The seven-model comparison gives a useful overview of performance across these settings.

  3. Appendix B includes a human study, controlled perturbations, and checks on hyperparameters and ranking stability. These provide evidence that the metrics respond to the errors they are intended to measure.

Critical weaknesses
  1. My main concern is how to interpret the scores when each trial has only one real execution as its reference. Repeated executions from similar initial observations may differ even with the same actions, particularly for pouring and towel manipulation. Appendix I acknowledges this, but does not quantify the variation. Without that comparison, it is hard to judge how much of the discrepancy comes from model error and how much from variation in the real executions. A small repeated-execution study would help establish whether this affects the reported gaps between models.

  2. The key-area score combines masked PSNR, LPIPS, and V-JEPA similarity, but the paper does not show whether the combination is better than its individual components. The human study supports the score's agreement with people; it does not establish the benefit of combining the three metrics. Ignoring background-only corruption is also largely a consequence of masking. The relevant comparison is with the individual masked metrics on the same preference pairs. Changing component weights, as in Appendix B.4.2, does not answer this question. A whole-frame baseline would also show how much the masking itself contributes.

  3. The long-horizon curves in Appendix A.3 average scores over [0,T)[0,T). As TT grows, later interactions are added, so the change in score mixes accumulated prediction errors with differences in event difficulty. The task and object/arm breakdowns are helpful, but do not resolve this issue. Testing the same later action segment after resetting to a real observation would help separate these effects.

  4. There is a mismatch in the fruit-task prompts. The authoring prompt in Appendix F.1 (p. 28) lists only three states: on the table, in a container, or not visible. The judging prompt and Appendix G.1 also include being grasped by the gripper. Grasp outcomes are central to the failure cases being evaluated. Please clarify whether this is only an error in the written prompt or whether grasp states were omitted from the generated criteria.

Questions for authors
  1. How much do repeated real executions differ under nominally matched initial conditions and action sequences? A small study on the pouring and towel tasks would help put the model errors in context.

  2. How does the full key-area score compare with masked PSNR, masked LPIPS, and masked V-JEPA on the same human-preference pairs, keeping masks, crops, sampling, and aggregation fixed? Please include uncertainty in the differences in agreement.

  3. For the same later action segment, how does prediction quality change when the model starts from a real observation instead of continuing the full autoregressive rollout?

  4. Did the actual fruit-annotation prompt allow "grasped by the gripper"? How are these grasp states represented in the final criteria?

Reviewer_D6BGOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes WorldOracle, a benchmark for action-conditioned world models against paired real-world executions, assessing whether predictions faithfully reproduce the consequences of recorded actions. Experiments show that accurate robot-motion prediction does not guarantee correct physical outcomes, and prediction fidelity generally declines over longer rollouts. The proposed metrics are evaluated through comparisons with human judgments and controlled perturbations.

Critical strengths
  1. Paired real executions provide a meaningful reference for action-conditioned prediction. The benchmark directly tests whether a model reproduces the observed consequences of a recorded action sequence, including failed manipulation attempts, rather than merely generating plausible or successful behavior. The inclusion of multiple embodiments, interaction dynamics, and policy-generated trajectories makes this a substantive evaluation resource.

  2. The evaluation framework is supported by substantial validation. The distinction between physical outcomes, spatial alignment, and local fidelity provides useful diagnostic information. The authors support the metrics with a human study involving 128 prediction pairs and 119 participants, reporting high agreement. These analyses provide stronger evidence than illustrative examples alone that the metrics capture relevant differences between predictions.

Critical weaknesses
  1. Insufficiently established incremental value over broader interactive world-model evaluation.

Although WorldOracle introduces physical correctness, spatial alignment, and area fidelity, existing interactive world-model benchmarks also assess related aspects of physical plausibility[1,2,6], spatial consistency[3,4,6], and visual fidelity[5,6] These underlying evaluation goals are not unique to embodied tasks, and the paper does not sufficiently demonstrate what distinctive diagnostic capabilities its metrics provide beyond existing assessments, despite its use of paired real-execution references.

  1. Unvalidated Correlation Between WorldOracle Scores and WAM Policy Performance WorldOracle measures prediction fidelity in embodied settings but does not validate how its scores relate to world-action model (WAM) policy performance. It therefore remains unclear whether these scores can meaningfully inform WAM selection or design.

  2. Uncalibrated variability under a single real-world reference. Each sample uses a single real-world execution as its reference, potentially penalizing plausible alternative outcomes arising from stochastic dynamics. Without repeated executions to quantify natural variability, the evaluation may conflate prediction errors with environmental randomness.

[1] WorldModelBench: Judging Video Generation Models As World Models, Li et al., 2025

[2] Genie 2: A Large-Scale Foundation World Model,Google DeepMind, 2024

[3] WorldScore: A Unified Evaluation Benchmark for World Generation, Duan et al., 2025

[4] VBench: Comprehensive Benchmark Suite for Video Generative Models, Huang et al., 2024

[5] WBENCH: A Comprehensive Multi-turn Benchmark for Interactive Video World Model Evaluation, Ying et al., 2026

[6] WorldRoamBench: An Open-World Benchmark for Long-Horizon Stability of Interactive World Models, Xu et al., 2026

Questions for authors
  1. Do higher WorldOracle scores predict better WAM policy performance, supporting their use for WAM selection or design?

  2. What additional diagnostic value do your metrics provide over existing physical-plausibility, spatial-consistency, and visual-fidelity assessments?

  3. How to distinguish prediction errors from natural execution variability when each sample has only one real-world reference?

Reviewer_Jph4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces WorldOracle, a benchmark for evaluating action-conditioned world models (ACWMs) against paired real-world executions. Given the recorded initial observation and low-level action sequence, a model generates an open-loop rollout, which is then compared with the real execution of the same actions. The authors collect a new AgiBot G2 training suite, use the public ABC-130K dataset for YAM, and evaluate human-hand models zero-shot. The evaluation suite consists of long human-operated episodes on the three embodiments and a policy-rollout split produced by a closed-loop π0.5 policy, covering four tasks (pick fruit, hang towel, pour water, pull drawer). Each episode is divided into multiple interaction segments with both successful and failed attempts.

Models are evaluated along four dimensions: physical state correctness (a VLM authors criteria from the real keyframe and judges whether the predicted keyframe satisfies them), interaction spatial alignment (displacement of DINOv3 feature matches within masks), key interaction area fidelity (PSNR, LPIPS and V-JEPA similarity within masked regions), and multi-view action coherence. The authors report a leaderboard of seven models and validate the metrics through a human agreement study, controlled perturbations and robustness analyses.

Critical strengths
  1. The problem is well motivated. The paper clearly separates "the rollout looks plausible" from "the rollout agrees with the real execution", and the latter is exactly what is needed when world models are used for policy evaluation or data generation. The example in Fig. 1, where the real gripper misses the fruit but the model generates a successful grasp, makes this point clearly. Organizing the evaluation around interaction segments with successful and failed outcomes is also a suitable design for this question.

  2. The geometric and appearance metrics focus on interaction regions and are validated in a targeted way. Spatial alignment and key-area fidelity are computed only within the regions of the manipulated objects and the robot arms (or hands), and objects and arms are reported separately, which avoids whole-frame metrics being dominated by the background. The controlled experiments show that key-area fidelity is unchanged when only the background is corrupted, and that only multi-view coherence drops when a single auxiliary view is delayed, indicating that these metrics respond selectively to the properties they are designed to measure.

Critical weaknesses
  1. The central claim of "causal obedience" is not directly tested. The authors list "following the given actions even when they are incorrect or the attempt fails" as the first of three stress tests, and the evaluation suite deliberately increases the proportion of failed segments. However, all results are broken down only by task, entity, horizon and embodiment; outcomes of successful and failed segments are never analyzed separately.
  • The two types of error have different costs. Predicting a real failure as a success makes policy evaluation over-optimistic and gives generated data wrong labels, whereas predicting a real success as a failure is much less harmful. Results should therefore be reported by real outcome × predicted outcome.
  • The physical-state criteria describe only the object states in the real keyframe. For example, the pick-fruit criteria are restricted to the locations of the fruits, so for an empty-grasp failure the criterion would simply be "the fruit is on the table". A model that does not execute the actions at all would then receive full credit on such segments.
  • There is no control that tests whether the models actually use the actions, e.g., starting from the same initial observation but conditioning on actions from another episode or on shuffled actions, and checking whether the metrics drop clearly. Without such a control, one cannot distinguish "following the actions" from "generating the typical outcome of the scene".
  • Related work has examined similar issues. MiraBench (Yang et al., 2026, arXiv:2605.29360) defines predicting success under failure-inducing actions as optimism bias, but its failure outcomes are assumed from the perturbation type rather than obtained by real execution. "Hallucination in World Models is Predictable and Preventable" (Hansen and Wang, 2026, arXiv:2606.27326) detects action-insensitive predictions by shuffling actions within a batch. With failed segments obtained from real executions, this paper is in the best position to carry out such analyses.
  • The introduction states that the evaluation covers counterfactual behaviors, but all actions in the evaluation suite were actually executed on the real robot; there are no counterfactual actions.
  1. The leaderboard cannot separate the models themselves from the authors' adaptation of them.
  • Training steps and batch sizes differ substantially across models, the rule for selecting evaluation checkpoints is not described, and the action interfaces of several models were modified.
  • There is no evidence that the adaptation succeeded, such as performance on held-out data from the training distribution, or zero-shot results for models whose action interfaces are compatible.
  • The pretraining data of some models already include AgiBot, which is the main evaluation embodiment, so it is unclear to what extent the ranking reflects how close each model's pretraining embodiment is to the evaluation embodiment.
  • The paper does not show any generated results labeled by model, and the supplementary material contains only code without videos. The qualitative examples in Fig. 1 and Fig. 4 each show a single frame without the initial frame, the time step or the model name, so readers cannot tell whether a model failed to execute the action or executed it incorrectly, nor verify that the adapted models work properly.
Questions for authors

Questions that could affect my assessment:

  1. Could you report each model's performance separately by the real and predicted outcome of each segment, in particular the rate at which real failures are predicted as successes? Could you add control experiments with replaced or shuffled actions?
  2. What common protocol was used to adapt the models, and how were the evaluation checkpoints selected? Could you report the performance of the adapted models on held-out data from the training distribution, and zero-shot results where feasible? Could you provide generated videos for each model, or at least consecutive frames with the initial frame and time steps?
  3. Physical state correctness relies entirely on the same VLM for both authoring and judging the criteria. What were the results of the human verification of the criteria? Could you report the per-criterion agreement between the VLM and human annotators?
  4. How were the hand poses fed to the models in the human-hand split obtained? If they were estimated from the real videos being predicted, the input itself carries information about the reference.
  5. Different models output videos at different frame rates, and the human-hand data also differ from the robot data in frame rate. How are the generated videos temporally aligned with the real recordings, and how are the outcome keyframes matched?

Clarifications and suggestions (not affecting my rating):

  1. The paper treats cross-embodiment generalization as part of being "reliable". Cross-embodiment generalization seems to be a higher-level requirement that becomes relevant only after in-domain coherence is achieved, and the unequal adaptation settings across the three embodiments in this paper cannot support such a comparison anyway. By contrast, switching from human teleoperation to policy-generated actions on the same embodiment is closer to reliability itself, but the evidence for the policy-rollout split is limited: the episodes are short, the failure ratio is not reported, and there is no analysis of how policy actions differ from human-operated ones.
  2. The importance of temporal endurance depends on the intended use of the model. If it is kept, I suggest re-initializing each segment from the real frame at its start and evaluating segments separately, to distinguish error accumulation from the intrinsic difficulty of individual segments.
  3. Each episode has only one real execution as the reference. I suggest repeating the same action sequence from the same initial state on at least a small subset of episodes, to quantify the variability of real executions themselves and judge whether the score differences between models are meaningful.
  4. Object alignment and fidelity are consistently lower than those of the arms. To what extent does this come from the metrics themselves? Objects are usually smaller, and scenes often contain objects with similar appearance, which makes matching over the whole feature map more error-prone.
  5. The annotations in Fig. 1 do not distinguish whether the outcome judgment is correct from whether the prediction is accurate. In the hang-towel example, the label says the grasp succeeds; this outcome judgment may agree with the real execution, but the position and shape of the towel and the poses of the arms in the predicted frame differ clearly from the real execution. I suggest marking these two aspects separately in the figure.
  6. Fig. 2(d) contains only icons and names and conveys little information. The claim that pick fruit is the hardest task for the models on the policy-rollout split is not fully consistent with Fig. S2. Please also clarify how the "40 events × 7 models" pairs in the robustness study are formed, and how the ranking correlation of the multi-view metric is computed when only two models are evaluated.

Comments on the paper

1 comment

Dear authors and reviewers,

The author-reviewer discussion period closes on November 18 (AoE). Please engage in response periods so that we have our issues discussed thoroughly.

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.