- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
WorldOracle evaluates action-conditioned world models against recorded real-world executions, using the same initial observation and low-level action sequence. The paper introduces an AgiBot G2 training set and 335 evaluation episodes spanning three embodiments and four tasks across the suite, with human-operated and policy-generated actions. Seven models are evaluated using four metrics for physical outcomes, spatial alignment, interaction-region fidelity, and multi-view alignment against real references. The results show performance degradation over long rollouts and a gap between following robot motion and predicting object interactions.
Critical strengths
-
Pairing recorded low-level actions with real executions gives the benchmark a clear purpose. The failed attempts are particularly useful: they expose cases where a model predicts a successful interaction despite actions that failed in reality.
-
The suite covers several interaction types, three embodiments, long human-operated episodes, and a separate policy-rollout split. The seven-model comparison gives a useful overview of performance across these settings.
-
Appendix B includes a human study, controlled perturbations, and checks on hyperparameters and ranking stability. These provide evidence that the metrics respond to the errors they are intended to measure.
Critical weaknesses
-
My main concern is how to interpret the scores when each trial has only one real execution as its reference. Repeated executions from similar initial observations may differ even with the same actions, particularly for pouring and towel manipulation. Appendix I acknowledges this, but does not quantify the variation. Without that comparison, it is hard to judge how much of the discrepancy comes from model error and how much from variation in the real executions. A small repeated-execution study would help establish whether this affects the reported gaps between models.
-
The key-area score combines masked PSNR, LPIPS, and V-JEPA similarity, but the paper does not show whether the combination is better than its individual components. The human study supports the score's agreement with people; it does not establish the benefit of combining the three metrics. Ignoring background-only corruption is also largely a consequence of masking. The relevant comparison is with the individual masked metrics on the same preference pairs. Changing component weights, as in Appendix B.4.2, does not answer this question. A whole-frame baseline would also show how much the masking itself contributes.
-
The long-horizon curves in Appendix A.3 average scores over . As grows, later interactions are added, so the change in score mixes accumulated prediction errors with differences in event difficulty. The task and object/arm breakdowns are helpful, but do not resolve this issue. Testing the same later action segment after resetting to a real observation would help separate these effects.
-
There is a mismatch in the fruit-task prompts. The authoring prompt in Appendix F.1 (p. 28) lists only three states: on the table, in a container, or not visible. The judging prompt and Appendix G.1 also include being grasped by the gripper. Grasp outcomes are central to the failure cases being evaluated. Please clarify whether this is only an error in the written prompt or whether grasp states were omitted from the generated criteria.
Questions for authors
-
How much do repeated real executions differ under nominally matched initial conditions and action sequences? A small study on the pouring and towel tasks would help put the model errors in context.
-
How does the full key-area score compare with masked PSNR, masked LPIPS, and masked V-JEPA on the same human-preference pairs, keeping masks, crops, sampling, and aggregation fixed? Please include uncertainty in the differences in agreement.
-
For the same later action segment, how does prediction quality change when the model starts from a real observation instead of continuing the full autoregressive rollout?
-
Did the actual fruit-annotation prompt allow "grasped by the gripper"? How are these grasp states represented in the final criteria?
Dear authors and reviewers,
The author-reviewer discussion period closes on November 18 (AoE). Please engage in response periods so that we have our issues discussed thoroughly.
AC