- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces Phys-WorldBench, a benchmark evaluating embodied world models across passive prediction, action-conditioned prediction, and robot action generation. It contains 4,500 real-robot trajectories across 50 tasks, including physically executed counterfactual failures. The experiments reveal moderate correlations between adjacent evaluation levels, but weak correlations between passive prediction and real-robot performance.
Critical strengths
Real-World Counterfactual Evaluation: The benchmark provides real execution outcomes for both successful and failed actions, allowing more direct evaluation of action-conditioned predictions. Comprehensive Evaluation: The three-level framework, diverse manipulation tasks, and extensive real-robot experiments provide useful insights into the relationship between world prediction and robot execution.
Critical weaknesses
The three levels use different models and training setups, making it difficult to determine whether the observed correlations truly reflect capability transfer. The Level-2 robustness score measures changes in prediction quality rather than absolute counterfactual accuracy. Its percentile-based normalization also makes the results dependent on the evaluated model set.
Questions for authors
Section 5.1 suggests that factual fidelity uses both GT and CF samples, while Appendix C.2 states that only GT samples contribute. Could the authors clarify this? The Level-2 trajectory metric mainly tracks gripper motion. How does the evaluation distinguish correct robot motion from correct physical effects on manipulated objects? Could the authors add a simple action-shuffling ablation for Level 2? This would help verify whether the models actually use the action inputs when predicting future states.
5 comments
Response to Reviewer fWuV W1 (2/3): Pool-Free Normalization Stability
-
Pool-Free Fixed-Anchor Normalization and Multi-Scheme Stability : In Level 2, we initially used non-parametric percentile ranks across the 2-model pool (Eq. 15) where all three statistics are oriented so that larger positive values indicate worse interventional performance (larger
D_Wass, or larger positive degradation inD_gapandD_tail), whereas Level 1 (Appendix C.1) uses frozen empirical reference bounds to linearly map raw metrics onto[0, 1]. Adopting the same monotonic orientation and frozen-bound design for Level 2 (Fixed-Anchor Normalization—linearly mapping empirical boundsD_Wass in [0.02, 0.55], signedD_gap in [-0.65, 0.10], and signedD_tail in [-0.30, 0.10]onto[0, 1]via1 - clip((x - a_k) / (b_k - a_k), 0, 1)) unifies our Level-1 and Level-2 protocols and allows any future model to be scored standalone without re-ranking against a model pool. We compared five normalization schemes under the Table 4 protocol and confirmed consistent model ordering and task-rank stability:Normalization Scheme Pool-Free? MotusR /L2-PWSCosmos2.5R /L2-PWSSpearman rho (R / L2-PWS)2-Model Percentile (Table 4) No 0.5316 / 0.4963 0.4731 / 0.4686 1.000 / 1.000 3-Model Percentile (+ FLUX)No 0.6127 / 0.5369 0.5631 / 0.5136 0.951 / 0.974 Min–Max No 0.6340 / 0.5475 0.5929 / 0.5285 0.949 / 0.954 Z-Score ( Phi(-z))No 0.5533 / 0.5072 0.4918 / 0.4779 0.980 / 0.987 Fixed-Anchor Yes 0.6449 / 0.5530 0.5986 / 0.5313 0.953 / 0.969 Across all five schemes,
Motusremains higher thanCosmos2.5on bothRandL2-PWS, task-level Spearman rank correlations with the submitted scores range from0.949to0.987(p < 1e-38), and Fixed-AnchorL2-PWSretains a significant family-pooled correlation with L3 progress (all three model families: r = 0.319, p = 0.0005). We will add Fixed-Anchor scores alongsideF_CFand raw statistics in the revision, so future models can be evaluated independently of any normalization pool.
Response to Reviewer fWuV Q1 (3/3): Metric Clarifications, Object Tracking, and Action Shuffling
Q1.1: Clarifying Section 5.1 vs. Appendix C.2 on GT and CF Samples
Thank you for catching this wording mismatch. Our implementation (Table 4, column "Factual Effect Fidelity (GT)", and Figure 3) followed Appendix C.2 (Eq. 9) and computed F on GT samples only (F = F_GT), whereas "GT and CF samples" in Section 5.1 and Table 4's caption was an imprecise summary of Table 4 as a whole. As shown in W1 above, replacing F_GT with F_GT+CF preserves the same model ranking, virtually identical task-level L2-PWS scores (r = 0.980–0.984), and significant L2–L3 correlations (p <= 0.020). We will correct the wording in Section 5.1 and Table 4's caption in the revision.
Q1.2: Distinguishing Robot Motion from Physical Effects on Manipulated Objects
-
Scene and Object Terms: In the
Fformulation, gripper trajectory accuracy (M_traj) constitutes 1 of 4 equal-weighted dimensions (25% ofF, 12.5% ofL2-PWS), while the remaining 75% evaluates full-frame pixel fidelity (PSNR), spatiotemporal representations (V-JEPA), and VLM-judged Physics Adherence (M_phys, averaging Robot-Arm IntegrityM_armand Object ConsistencyM_obj). Table 4 already reflects this distinction:Motusattains higher gripper trajectory accuracy (0.2788vs.0.2252), whereasCosmos2.5attains higher PSNR (0.5869vs.0.4360) and V-JEPA similarity (0.6096vs.0.5784). -
Explicit Manipulated-Object Tracking and Outcome Evaluation:
To evaluate object-level physical outcomes directly, we used SAM3–CoTracker to target manipulated objects to measure object center trajectory accuracy and final-frame object mask IoU:
- Consistent with PSNR and V-JEPA in Table 4,
Cosmos2.5outperformsMotuson moving manipulated objects in both object trajectory accuracy (0.742vs.0.311) and final mask IoU (0.371vs.0.120), whereasMotusleads on gripper tracking (0.2788vs.0.2252). - Incorporating object trajectory accuracy and mask IoU into
L2-PWSyields nearly identical task-level scores (r = 0.96–0.98) and preserves significant L2–L3 correlations (Wan: r = 0.342, p = 0.031;Cosmos: r = 0.391, p = 0.014). We will add these explicit object metrics to revised Table 4.
- Consistent with PSNR and V-JEPA in Table 4,
Q1.3: Action-Shuffling and Action-Perturbation Ablations for Level 2
We tested whether Level-2 predictions depend on input actions using the same protocol for Cosmos2.5 (action) and Motus . We kept the initial frameo_t and instructiong fixed, then replaced each ground-truth action sequence with one from another episode of the same task.
| Model | Gripper Traj Acc Δ (%) | Traj Acc Wilcoxon p | Normalized PSNR Δ (%) | PSNR Wilcoxon p |
|---|---|---|---|---|
| Cosmos2.5(action) | -21.9% | 1.78e-15 | -12.0% | 1.78e-15 |
| Motus | -6.3% | 1.33e-5 | -2.6% | 1.76e-8 |
Both models show statistically significant drops in gripper trajectory accuracy and normalized PSNR after action shuffling, confirming that neither ignores the action input. Their sensitivity gap reflects their training objectives: Cosmos2.5 (action) is post-trained specifically for action-conditioned video prediction (-21.9% gripper accuracy, -12.0% PSNR), whereas Motus jointly trains video prediction and policy generation from (o_t, g), relying more on visual-language priors (-6.3% and -2.6%). This stronger action-following fidelity in Cosmos2.5 explains why it achieves higher F_CF and a smaller GT–CF trajectory-accuracy distribution shift (D_Wass) in W1—as it faithfully executes counterfactual actions rather than falling back on goal-conditioned (o_t, g) priors, which causes Motus to deviate from true CF trajectories (lower F_CF and larger D_Wass). Meanwhile, Motus leads on R via smaller signed nDTW degradation (D_gap, D_tail) relative to its own GT baseline. We will add this ablation to the revision.
Response to Reviewer fWuV W1 (1/3): Cross-Level Evidence and Counterfactual Fidelity
We sincerely thank the reviewer for this thoughtful comment. Below, we address the three points raised in
W1in order:Controlled Cross-Level Evaluations: As summarized in the General Response, we evaluated the exact same fine-tuned
Motuscheckpoint across L1, L2 and L3 to eliminate cross-level model differences. We also evaluated a new model familyFLUX-3-action.Across all four settings, the adjacent levels (
L1–L2andL2–L3) remain significantly correlated whileL1–L3remains non-significant, suggesting that this is a relatively general and robust pattern.Design Rationale of
R& Standalone Counterfactual Fidelity : By design,L2-PWS = (F + R) / 2decouples absolute prediction fidelity (F) from interventional robustness (R), which combines unsigned trajectory-accuracy distribution proximity (D_Wass, smaller is better) with signed directional non-degradation gaps (D_gap,D_tail, penalizing positive degradation when CF predictions degrade relative to GT; Appendix C.2 Eqs. 10–15). This avoids double-counting baseline generation quality in both terms while ensuring that a model with poor predictions on both GT and CF receives a lowFand thus a lowL2-PWS. Consistent with Table 4—whereCosmos2.5is already slightly higher onF(0.4640vs.0.4610) whileMotusleads onR(0.5316vs.0.4731) and therefore on overallL2-PWS(0.4963vs.0.4686)—incorporating CF into fidelity asF_GT+CFpreserves the same ranking (Motus 0.4735 > Cosmos2.5 0.4622):F_GTF_CFF_GT+CFRL2-PWS(F_GT)L2-PWS(F_GT+CF)MotusCosmos2.5Thus the two models show complementary profiles:
Cosmos2.5attains slightly higher absolute fidelity (F/F_CF) and smaller trajectory-accuracy distribution shift (D_Wass= 0.143 vs. 0.201), whereasMotusavoids positive temporal-alignment degradation relative to its GT baseline (D_gap,D_tail) and therefore leads onRand compositeL2-PWSunder eitherF_GTorF_GT+CF. Task-level scores underF_GT+CFstay virtually identical to the submittedL2-PWS(r = 0.980–0.984) and preserve significant L2–L3 correlations (Wan: r = 0.367, p = 0.020;Cosmos: r = 0.385, p = 0.016). We will includeF_CF,F_GT+CF, and the unnormalized raw statistics as additional reference columns in revised Table 4 and Appendix C.2.