acceptodds
Under review as a conference paper at ICLR 2027

Phys-WorldBench: A Unified Benchmark for Evaluating Embodied World Models from Passive Prediction to Active Intervention

Abstract

Embodied intelligence requires model to understand how actions transform the physical world. World models provide a natural foundation for this ability, yet their physical interaction capabilities remain insufficiently characterized. Existing benchmarks often evaluate passive prediction or action-conditioned modeling separately, lacking a unified protocol that connects world prediction, action-effect modeling, and real-world action generation. They also commonly organize evaluation around semantic tasks, making it difficult to cover the underlying mechanisms of physical interaction. We introduce , a unified real-world benchmark that organizes tabletop manipulation through interaction skills and physical constraints. Its three-level protocol spans passive future prediction, action-conditioned world modeling, and world action modeling within a shared physical interaction space. Built on 4,500 real-robot trajectories (30 hours), Phys-WorldBench covers 50 atomic, compositional, and out-of-distribution tasks. Across representative world models and embodied policies, we observe substantial variation across the three levels, with persistent challenges on compositional and out-of-distribution tasks. Visually plausible prediction does not consistently translate into accurate action-effect modeling or robust real-world execution. Cross-level analysis further finds significant moderate associations between adjacent levels (– for L1–L2 and – for L2–L3, all across Wan and Cosmos), while passive prediction remains weakly associated with real-robot progress. Together, these results suggest that predictive quality alone provides an incomplete measure of embodied world-model capability, motivating evaluation across the full progression from world prediction to physical interaction.

Primary area:
datasets and benchmarks
Keywords:
Embodied World Models, Physical Interaction, Robot Manipulation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_fWuVOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Phys-WorldBench, a benchmark evaluating embodied world models across passive prediction, action-conditioned prediction, and robot action generation. It contains 4,500 real-robot trajectories across 50 tasks, including physically executed counterfactual failures. The experiments reveal moderate correlations between adjacent evaluation levels, but weak correlations between passive prediction and real-robot performance.

Critical strengths

Real-World Counterfactual Evaluation: The benchmark provides real execution outcomes for both successful and failed actions, allowing more direct evaluation of action-conditioned predictions. Comprehensive Evaluation: The three-level framework, diverse manipulation tasks, and extensive real-robot experiments provide useful insights into the relationship between world prediction and robot execution.

Critical weaknesses

The three levels use different models and training setups, making it difficult to determine whether the observed correlations truly reflect capability transfer. The Level-2 robustness score measures changes in prediction quality rather than absolute counterfactual accuracy. Its percentile-based normalization also makes the results dependent on the evaluated model set.

Questions for authors

Section 5.1 suggests that factual fidelity uses both GT and CF samples, while Appendix C.2 states that only GT samples contribute. Could the authors clarify this? The Level-2 trajectory metric mainly tracks gripper motion. How does the evaluation distinguish correct robot motion from correct physical effects on manipulated objects? Could the authors add a simple action-shuffling ablation for Level 2? This would help verify whether the models actually use the action inputs when predicting future states.

5 comments
AuthorsOpenReview ↗

Response to Reviewer fWuV W1 (1/3): Cross-Level Evidence and Counterfactual Fidelity

We sincerely thank the reviewer for this thoughtful comment. Below, we address the three points raised inW1 in order:

  1. Controlled Cross-Level Evaluations: As summarized in the General Response, we evaluated the exact same fine-tuned Motuscheckpoint across L1, L2 and L3 to eliminate cross-level model differences. We also evaluated a new model family FLUX-3-action.

    Evaluation settingControl levelL1–L2 r (p)L2–L3 r (p)L1–L3 r (p)
    MotusIdentical weights0.359 (0.023)0.328 (0.039)0.252 (0.116)
    Cosmos FamilyShared initialization and training data0.343 (0.033)0.378 (0.018)0.118 (0.474)
    Wan FamilyBackbone-aligned family0.359 (0.023)0.332 (0.036)0.099 (0.542)
    FLUX-3-actionShared initialization and training data0.439 (0.005)0.315 (0.048)0.247 (0.125)

    Across all four settings, the adjacent levels (L1–L2and L2–L3) remain significantly correlated while L1–L3 remains non-significant, suggesting that this is a relatively general and robust pattern.

  2. Design Rationale of R & Standalone Counterfactual Fidelity : By design, L2-PWS = (F + R) / 2 decouples absolute prediction fidelity (F) from interventional robustness (R), which combines unsigned trajectory-accuracy distribution proximity (D_Wass, smaller is better) with signed directional non-degradation gaps (D_gap, D_tail, penalizing positive degradation when CF predictions degrade relative to GT; Appendix C.2 Eqs. 10–15). This avoids double-counting baseline generation quality in both terms while ensuring that a model with poor predictions on both GT and CF receives a low F and thus a low L2-PWS. Consistent with Table 4—where Cosmos2.5 is already slightly higher on F (0.4640 vs. 0.4610) while Motus leads on R (0.5316 vs. 0.4731) and therefore on overall L2-PWS (0.4963 vs. 0.4686)—incorporating CF into fidelity as F_GT+CF preserves the same ranking (Motus 0.4735 > Cosmos2.5 0.4622):

    ModelF_GTF_CFF_GT+CFRL2-PWS(F_GT)L2-PWS(F_GT+CF)
    Motus0.46100.36960.41530.53160.49630.4735
    Cosmos2.50.46400.43860.45130.47310.46860.4622

    Thus the two models show complementary profiles: Cosmos2.5 attains slightly higher absolute fidelity (F / F_CF) and smaller trajectory-accuracy distribution shift (D_Wass = 0.143 vs. 0.201), whereas Motus avoids positive temporal-alignment degradation relative to its GT baseline (D_gap, D_tail) and therefore leads on R and composite L2-PWS under either F_GT or F_GT+CF. Task-level scores under F_GT+CF stay virtually identical to the submitted L2-PWS (r = 0.980–0.984) and preserve significant L2–L3 correlations (Wan: r = 0.367, p = 0.020; Cosmos: r = 0.385, p = 0.016). We will include F_CF, F_GT+CF, and the unnormalized raw statistics as additional reference columns in revised Table 4 and Appendix C.2.

AuthorsOpenReview ↗

Response to Reviewer fWuV W1 (2/3): Pool-Free Normalization Stability

  1. Pool-Free Fixed-Anchor Normalization and Multi-Scheme Stability : In Level 2, we initially used non-parametric percentile ranks across the 2-model pool (Eq. 15) where all three statistics are oriented so that larger positive values indicate worse interventional performance (larger D_Wass, or larger positive degradation in D_gap and D_tail), whereas Level 1 (Appendix C.1) uses frozen empirical reference bounds to linearly map raw metrics onto [0, 1]. Adopting the same monotonic orientation and frozen-bound design for Level 2 (Fixed-Anchor Normalization—linearly mapping empirical bounds D_Wass in [0.02, 0.55], signed D_gap in [-0.65, 0.10], and signed D_tail in [-0.30, 0.10] onto [0, 1] via 1 - clip((x - a_k) / (b_k - a_k), 0, 1)) unifies our Level-1 and Level-2 protocols and allows any future model to be scored standalone without re-ranking against a model pool. We compared five normalization schemes under the Table 4 protocol and confirmed consistent model ordering and task-rank stability:

    Normalization SchemePool-Free?Motus R / L2-PWSCosmos2.5 R / L2-PWSSpearman rho (R / L2-PWS)
    2-Model Percentile (Table 4)No0.5316 / 0.49630.4731 / 0.46861.000 / 1.000
    3-Model Percentile (+FLUX)No0.6127 / 0.53690.5631 / 0.51360.951 / 0.974
    Min–MaxNo0.6340 / 0.54750.5929 / 0.52850.949 / 0.954
    Z-Score (Phi(-z))No0.5533 / 0.50720.4918 / 0.47790.980 / 0.987
    Fixed-AnchorYes0.6449 / 0.55300.5986 / 0.53130.953 / 0.969

    Across all five schemes, Motus remains higher than Cosmos2.5 on both R and L2-PWS, task-level Spearman rank correlations with the submitted scores range from 0.949 to 0.987 (p < 1e-38), and Fixed-Anchor L2-PWS retains a significant family-pooled correlation with L3 progress (all three model families: r = 0.319, p = 0.0005). We will add Fixed-Anchor scores alongside F_CF and raw statistics in the revision, so future models can be evaluated independently of any normalization pool.

AuthorsOpenReview ↗

Response to Reviewer fWuV Q1 (3/3): Metric Clarifications, Object Tracking, and Action Shuffling

Q1.1: Clarifying Section 5.1 vs. Appendix C.2 on GT and CF Samples

Thank you for catching this wording mismatch. Our implementation (Table 4, column "Factual Effect Fidelity (GT)", and Figure 3) followed Appendix C.2 (Eq. 9) and computed F on GT samples only (F = F_GT), whereas "GT and CF samples" in Section 5.1 and Table 4's caption was an imprecise summary of Table 4 as a whole. As shown in W1 above, replacing F_GT with F_GT+CF preserves the same model ranking, virtually identical task-level L2-PWS scores (r = 0.980–0.984), and significant L2–L3 correlations (p <= 0.020). We will correct the wording in Section 5.1 and Table 4's caption in the revision.

Q1.2: Distinguishing Robot Motion from Physical Effects on Manipulated Objects

  1. Scene and Object Terms: In the F formulation, gripper trajectory accuracy (M_traj) constitutes 1 of 4 equal-weighted dimensions (25% of F, 12.5% of L2-PWS), while the remaining 75% evaluates full-frame pixel fidelity (PSNR), spatiotemporal representations (V-JEPA), and VLM-judged Physics Adherence (M_phys, averaging Robot-Arm Integrity M_arm and Object Consistency M_obj). Table 4 already reflects this distinction: Motus attains higher gripper trajectory accuracy (0.2788 vs. 0.2252), whereas Cosmos2.5 attains higher PSNR (0.5869 vs. 0.4360) and V-JEPA similarity (0.6096 vs. 0.5784).

  2. Explicit Manipulated-Object Tracking and Outcome Evaluation:

    To evaluate object-level physical outcomes directly, we used SAM3–CoTracker to target manipulated objects to measure object center trajectory accuracy and final-frame object mask IoU:

    • Consistent with PSNR and V-JEPA in Table 4, Cosmos2.5 outperforms Motus on moving manipulated objects in both object trajectory accuracy (0.742 vs. 0.311) and final mask IoU (0.371 vs. 0.120), whereas Motus leads on gripper tracking (0.2788 vs. 0.2252).
    • Incorporating object trajectory accuracy and mask IoU into L2-PWS yields nearly identical task-level scores (r = 0.96–0.98) and preserves significant L2–L3 correlations (Wan: r = 0.342, p = 0.031; Cosmos: r = 0.391, p = 0.014). We will add these explicit object metrics to revised Table 4.

Q1.3: Action-Shuffling and Action-Perturbation Ablations for Level 2

We tested whether Level-2 predictions depend on input actions using the same protocol for Cosmos2.5 (action) and Motus . We kept the initial frameo_t and instructiong fixed, then replaced each ground-truth action sequence with one from another episode of the same task.

ModelGripper Traj Acc Δ (%)Traj Acc Wilcoxon pNormalized PSNR Δ (%)PSNR Wilcoxon p
Cosmos2.5(action)-21.9%1.78e-15-12.0%1.78e-15
Motus-6.3%1.33e-5-2.6%1.76e-8

Both models show statistically significant drops in gripper trajectory accuracy and normalized PSNR after action shuffling, confirming that neither ignores the action input. Their sensitivity gap reflects their training objectives: Cosmos2.5 (action) is post-trained specifically for action-conditioned video prediction (-21.9% gripper accuracy, -12.0% PSNR), whereas Motus jointly trains video prediction and policy generation from (o_t, g), relying more on visual-language priors (-6.3% and -2.6%). This stronger action-following fidelity in Cosmos2.5 explains why it achieves higher F_CF and a smaller GT–CF trajectory-accuracy distribution shift (D_Wass) in W1—as it faithfully executes counterfactual actions rather than falling back on goal-conditioned (o_t, g) priors, which causes Motus to deviate from true CF trajectories (lower F_CF and larger D_Wass). Meanwhile, Motus leads on R via smaller signed nDTW degradation (D_gap, D_tail) relative to its own GT baseline. We will add this ablation to the revision.

Reviewer_GypAOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors proposes Phys-WorldBench, a unified real-world benchmark for evaluating embodied world models on 3 levels: level 1 passive world modeling, level 2 action-conditioned world modeling, and level 3 world action modeling. Built on 4,500 real-robot trajectories (~30 hours), Phys-WorldBench covers 50 atomic, compositional, and out-of-distribution tasks. Across evaluation of representative models, this paper observes substantial variation across the three levels.

Critical strengths
  1. A strong unified evaluation framework with multiple evaluation aspects. The three-level design provides a useful framework. I like the idea of evaluating passive prediction, action-conditioned prediction, and action generation in the same physical task space. These abilities are often evaluated separately, so this benchmark gives a more complete view of where current world models succeed and fail.

  2. The real counterfactual trajectories are useful. For level 2, the authors do not only use successful demonstrations. They also collect failed action trajectories. This is important because it tests whether a model follows the given action, instead of simply generating a plausible or successful future observations.

Critical weaknesses
  1. The cross-level analysis is based on too few model families. One of the main conclusions is that performance does not transfer directly from passive prediction to action-conditioned prediction and then to real-world execution. However, this analysis is mainly based on only two model families, Wan and Cosmos. Also, the models compared across levels are not controlled variants of the same system. They differ in architecture, pretraining, and training setup. Because of this, it is hard to know whether the observed correlations reflect a general property of world models, or just the behavior of these particular model families.

  2. The GT and counterfactual trajectories do not start from exactly the same state. The counterfactual design is interesting, but GT and counterfactual trajectories are collected in separate real-world runs. Their initial states I think may not be accurately matched, so there can still be differences in object position and appearance. So this makes it possible that some of the gap comes from the differences rather than the changed action.

Questions for authors
  1. Can the authors show whether cross-level trend also holds for another model family?

  2. Can the authors measure how much initial-state mismatch affects the GT-CF results?

5 comments
AuthorsOpenReview ↗

Response to Reviewer GypA W1&Q1 (1/2): Controlled Model Families and Third Model Family

Summary: Evaluating an identical single-system checkpoint (Motus) across all three levels and extending the three-level protocol to a third independent model family (FLUX-3-action) consistently reproduces the same cross-level trend observed in the submitted Wan and Cosmos family pairings: monotonic degradation across task splits, significant L1–L2 and L2–L3associations, and weak/non-significant L1–L3 associations.

Controlled Cross-Level Variants Alongside Submitted Wan and Cosmos Pairings (W1)

Among the current open-source models, only Wan and Cosmos officially release model suites spanning from L1 to L3 (other video backbones such as LTX-Video and CogVideoX lack official action/policy variants, whereas recent embodied models such as Gigaworld-0 in Table 3 or Genie Envisioner build upon Wan or Cosmos backbones). To eliminate cross-level confounding and expand family coverage, we evaluated the identical fine-tuned Motus checkpoint across L1, L2 and L3:

Evaluation settingControl levelL1–L2 r (p)L2–L3 r (p)L1–L3 r (p)
MotusIdentical weights0.359 (0.023)0.328 (0.039)0.252 (0.116)
Cosmos FamilyShared initialization and training data0.343 (0.033)0.378 (0.018)0.118 (0.474)
Wan FamilyBackbone-aligned family0.359 (0.023)0.332 (0.036)0.099 (0.542)
FLUX-3-actionShared initialization and training data0.439 (0.005)0.315 (0.048)0.247 (0.125)

Based on the results of the same model checkpoints Motus, we can also find adjacent levels (L1–L2 and L2–L3) remain significantly correlated while L1–L3 remains non-significant.

Adapting and Evaluating a new model family FLUX-3-action (Q1)

We also use FLUX-3-action as a third independent family, alongside current results of Wan and Cosmos.

  • Model Adaptation: Because FLUX-3-action does not release separate L1 or L2 checkpoints, we decoupled its video–text DiT backbone for L1 Passive World Modeling and adapted its 14-DoF bimanual action-conditioning and joint-policy interfaces on our training dataset for L2 Action-Conditioned World Modeling and L3 World Action Modeling.

  • Cross-Level Results on FLUX-3-action: Across the evaluation tasks with full GT and CF pairs, the third model family shows consistent performance degradation from Atomic (L1 = 0.425, L2-PWS = 0.479, L3 = 0.437) to Compositional (0.372, 0.467, 0.311) tasks, along with the same cross-level association pattern: adjacent levels are significantly correlated (L1–L2: r = 0.439, p = 0.005; L2–L3: r = 0.315, p = 0.048), whereas L1–L3 is not (r = 0.247, p = 0.125).

AuthorsOpenReview ↗

Response to Reviewer GypA W2&Q2 (2/2): Initial-State Confound Checks

We thank the reviewer for highlighting this potential factor (W2). To test how run-to-run variation in initial object placement and visual appearance affects our Level-2 evaluation (Q2), we ran three quantitative checks on Motus, Cosmos2.5 and FLUX-3-action:

  1. GT-vs-GT Split-Half Reset Baseline vs. GT-vs-CF Interventional Shift:

    We randomly split each evaluation case's 10 GT trajectories into two halves (200 random 5-vs-5 splits across evaluation cases) to measure the case-level baseline distributional shift caused solely by physical scene resets without any counterfactual action change. Across all three model families, the case-level GT-vs-CF distributional shift exceeds this GT-vs-GT reset baseline by 1.35x–3.19x:

    • Trajectory-Accuracy Wasserstein Distance (D_Wass, case-level): Motus 0.198 vs. 0.090 (2.21x, Wilcoxon p = 3.97e-7); Cosmos2.5 0.146 vs. 0.109 (1.35x, p = 0.0172); FLUX-3-action 0.210 vs. 0.096 (2.20x, p = 9.06e-9).
    • Mean nDTW Gap Magnitude (|D_gap|, case-level): Motus 0.316 vs. 0.099 (3.19x, p = 8.00e-10); Cosmos2.5 0.243 vs. 0.096 (2.53x, p = 1.13e-6); FLUX-3-action 0.204 vs. 0.077 (2.66x, p = 2.92e-9).
  2. First-Frame GT–CF SSIM Regression and High-SSIM Subset Filtering:

    We used paired GT–CF first-frame SSIM to test whether initial visual mismatch inflates the GT–CF gap:

    • Correlation check: If initial visual mismatch drove the GT–CF gap, tasks with lower first-frame SSIM (larger initial visual discrepancy) would show larger GT–CF trajectory-accuracy distribution shifts (D_Wass). We found no significant association in any model family (Motus :r = -0.037, p = 0.824 ;Cosmos2.5 :r = -0.173, p = 0.299 ;FLUX-3-action :r = 0.184, p = 0.261 ).
    • High-SSIM filtering: If poorly aligned runs were inflating our GT–CF gaps, discarding lower-SSIM tasks and keeping only tightly matched initial scenes (SSIM >= 0.80, matching the 0.82 same-task training alignment in Appendix B.3 Table 8) would shrink D_Wass and alter Level-2 scores. Instead, restricting evaluation to SSIM >= 0.80 leaves task-level D_Wass magnitudes virtually unchanged across all three families (Motus: 0.201 on all tasks vs. 0.184; Cosmos2.5: 0.143 vs. 0.138; FLUX-3-action: 0.204 vs. 0.226), keeps mean L2-PWS within 0.013 of the full benchmark and preserves the exact same model ordering.
  3. Initial Manipulated-Object Centroid Offset: We segmented the target objects in the initial frames of paired GT/CF episodes with SAM3. The mean GT–CF centroid offset (0.187) closely matched the natural intra-GT variation across successful trials (0.176; ratio = 1.06x). Further, the initial object offsets were not positively correlated with trajectory-accuracy gaps, and the association was not statistically significant (r = -0.117, p = 0.140 ).

Together, these three checks confirm that initial-state variation does not drive the observed GT–CF gaps. We will add this quantitative validation to Appendix B.3 in the revision.

AuthorsOpenReview ↗

Gentle Follow-up on Our Rebuttal

Thank you again for your constructive feedback.

We would like to kindly follow up on our rebuttal responses, where we have provided additional experiments addressing both of your main concerns:

  • W1/Q1: Controlled cross-level evaluation using the identical Motus checkpoint, along with results from a third model family (FLUX-3-action).
  • W2/Q2: Quantitative analyses of initial-state mismatch, including GT-vs-GT controls, SSIM-based filtering, and object-position comparisons.

We would greatly appreciate it if you could take a look at these additional results and let us know whether they help address your concerns. We would be happy to provide any further clarification.

Reviewer_tAc1OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Phys-WorldBench, a real-robot benchmark that evaluates embodied world models through passive future prediction (L1), action-conditioned future prediction (L2), and executable action generation (L3). It contains 4,500 trajectories across 20 atomic tasks, 20 compositional tasks, and 10 held-out OOD tasks, organized around physical interaction skills. The dataset includes deliberately unsuccessful action sequences with recorded physical outcomes, allowing predictions under alternative actions to be compared with what actually happened. The experiments cover video-generation models, action-conditioned world models, and robot policies. Across the Wan and Cosmos families, the authors find moderate task-level correlations between adjacent levels and weaker associations between passive prediction and real-robot progress. These results motivate evaluating prediction and execution capabilities together.

Critical strengths
  1. The shared evaluation setting is useful. The benchmark brings passive prediction, action-conditioned modeling, and real-robot execution into a common task space. Examining the same physical interactions through these different interfaces makes it easier to compare capabilities and identify differences in performance.

  2. The real-robot dataset is a valuable resource. In particular, the alternative-action trajectories come with recorded physical outcomes, providing concrete reference futures for action-conditioned evaluation. The coverage of atomic, compositional, and OOD tasks also supports studying how models handle more complex and unfamiliar interactions.

Critical weaknesses
  1. It is not yet clear how much the composite L2 score adds. Section 7 reports positive correlations between L2 scores and L3 execution progress (r = 0.332 for Wan and 0.378 for Cosmos). These may partly reflect task difficulty: tasks that are easier to predict may also be easier to execute. The analysis does not show whether L2 provides additional information beyond basic task characteristics, such as task category and duration, or individual metrics such as gripper trajectory accuracy. Comparing the composite score with these simpler baselines would help establish its value for assessing execution capability. Without this comparison, the cross-level results remain largely descriptive.

  2. The interpretation of capability transfer needs clearer qualification. The cross-level analysis compares different systems within each backbone family—for example, Wan2.2, Motus, and FastWAM at L1–L3, respectively. The authors acknowledge in Sections 5.1 and 6.2 that architecture and training differences are not controlled. These differences also limit what the results can tell us about capability transfer: an observed gap between prediction and execution could arise from predictive modeling, policy learning, or the connection between them. L3 measures actual robot performance, but does not identify the source of that gap. A matched evaluation linking prediction and execution within the same system would support a stronger interpretation. With the current evidence, the conclusions should remain focused on performance relationships among the evaluated systems.

Questions for authors
  1. What evidence shows that the composite L2 score provides useful information about execution beyond task difficulty or the individual metrics that make up the score?

  2. Given the differences in architecture and training across levels, which conclusions about capability transfer do the authors consider supported by the current comparisons?

7 comments
AuthorsOpenReview ↗

Response to Reviewer tAc1 W1&Q1 (1/3): Incremental Value of L2-PWS

Summary: Across all three model families (Wan, Cosmos, FLUX-3-action), composite L2-PWS adds statistically significant explanatory power for L3 performance beyond every single-clip component metric (ΔR² = +0.088 to +0.116, p_OLS <= 0.0010) and beyond R alone (ΔR² = +0.036, p_OLS = 0.0353). It also remains statistically significant after controlling for gripper accuracy, task category + duration, all 5 physical task features (r = +0.226, p_OLS = 0.0175, p_boot = 0.0200, 95% CI [0.039, 0.392]) and the manipulation difficulty by independent VLA pi_0.5 performance (r = +0.270, p_OLS = 0.0033, p_boot = 0.0066, 95% CI [0.083, 0.436]).

Comparison Against Individual Component Metrics In L2-PWS = (F + R) / 2 (Eq. 4), Factual Effect Fidelity (F) and Interventional Robustness (R) are combined with fixed equal weights (0.5 / 0.5) specified a priori. We evaluate each constituent metric in Eq. (9) in the same three-family pooled OLS regression (Wan, Cosmos, FLUX-3-action):

Level-2 MetricPooled r (p_OLS)ΔR² from Adding L2-PWS (p_OLS)
Gripper Trajectory Accuracy (M_traj)+0.049 (0.6003)+0.113 (0.0002)
Normalized PSNR (M_pixel)+0.176 (0.0579)+0.088 (0.0010)
V-JEPA Similarity (M_latent)+0.049 (0.5961)+0.116 (0.0002)
VLM Physics Adherence (M_phys)+0.037 (0.6944)+0.116 (0.0002)
Factual Effect Fidelity (F)+0.074 (0.4306)+0.114 (0.0002)
Interventional Robustness (R)+0.290 (0.0017)+0.036 (0.0353)
Composite L2-PWS+0.340 (0.0002)—

F and R were paired a priori to evaluate the two intrinsic requirements of action-conditioned world modeling (Appendix C.2): predicting accurate state transitions under nominal goal-achieving demonstrations (F) and avoiding trajectory degradation when input actions shift from GT to CF (R). Neither alone suffices—standard demonstration metrics (M_traj, PSNR, M_latent, M_phys, F) do not test off-distribution action interventions, whereas R alone would allow a degenerate model that outputs static or uniformly poor frames on both GT and CF (yielding small distributional shifts and non-positive degradation gaps) to score spuriously high. Empirically, the three-family pooled table shows why combining both matters: individual single-clip metrics show no statistically significant correlation with L3 (r <= +0.176, p >= 0.058), whereas R (r = +0.290) and L2-PWS (r = +0.340) are strongly associated with L3 performance. Adding L2-PWS significantly increases explained L3 variance over every single-clip metric (ΔR² = +0.088 to +0.116, p_OLS <= 0.0010) and over R alone (ΔR² = +0.036, p_OLS = 0.0353).

AuthorsOpenReview ↗

Response to Reviewer tAc1 W1&Q1 (2/3): Task Controls and Independent Difficulty

Partial Correlations Controlling for Observable Task Features and Independent VLA (pi_0.5) Progress

To test whether the L2–L3 association is explained by observable task attributes or shared manipulation difficulty, we pool all three evaluated model families (Wan, Cosmos, and FLUX-3-action). We residualize both the L2 score and L3 performance on each control set (plus family indicators), correlate the resulting residuals and report both OLS p-values and 10,000-draw task-clustered bootstrap p-values and 95% confidence intervals:

  • Gripper Trajectory Accuracy: Tests whether L2-PWS adds predictive value beyond single-clip gripper tracking accuracy.
  • Task Category + Duration: Controls for the 4 physical interaction categories (Section 3.2) and mean trajectory duration.
  • All 5 Physical Task Features: Jointly controls for interaction category, trajectory duration, atomic vs. compositional split, unique skill count and cumulative end-effector path length (Appendix B.4).
  • Independent VLA (pi_0.5) Progress: Controls for per-task progress of pi_0.5, which shares no world-model components and serves as an external proxy for generic manipulation difficulty (explaining R² = 0.15–0.68 of L3 variance across families).
Control Variables (L2-PWS vs. L3)Pooled Partial rp_OLSp_boot [95% Cluster CI]Pooled ΔR²
None (Unadjusted)+0.3400.00020.0026 [0.125, 0.516]+0.116
+ Gripper Accuracy+0.3370.00020.0024 [0.118, 0.516]+0.113
+ Category & Duration+0.2550.00640.0150 [0.053, 0.429]+0.045
+ All 5 Task Features+0.2260.01750.0200 [0.039, 0.392]+0.032
+ pi_0.5 Progress+0.2700.00330.0066 [0.083, 0.436]+0.042

Key Takeaways ( W1 & Q1):

  1. Incremental value beyond observable task features: Across all three model families, L2-PWS remains statistically significant under both OLS and task-clustered bootstrap tests after controlling for gripper accuracy (p_boot = 0.0024), task category + duration (p_boot = 0.0150), and all 5 physical task features simultaneously (r = +0.226, p_OLS = 0.0175, p_boot = 0.0200, 95% CI [0.039, 0.392]). Thus, the L2–L3 association is not reducible to task category, duration, or trajectory kinematics.
  2. Partial correlation beyond pi_0.5 difficulty (Q1): Controlling for the independent VLA pi_0.5 reduces the pooled correlation (r = +0.340 -> +0.270), while remaining statistically significant (p_OLS = 0.0033, p_boot = 0.0066, 95% CI [0.083, 0.436], ΔR² = +0.042). This shows that while generic manipulation difficulty explains part of the cross-level relationship, a significant L2–L3 association persists beyond it. We will add these three-family controlled analyses to Section 7 and Appendix E.
AuthorsOpenReview ↗

Response to Reviewer tAc1 W2&Q2 (3/3): Controlled Cross-Level Relationships and Scope

Summary: The results of the exact same fine-tuned Motus checkpoint across L1, L2 and L3 aligns with the Wan and Cosmos family pairings and a third independent model family (FLUX-3-action), showing the same cross-level pattern: significant adjacent-level correlations (L1–L2, L2–L3) and non-significant L1–L3 correlations.

Matched Within-System and Shared-Initialization Evaluations (W2)

  • Identical-Checkpoint Single-System Evaluation (Motus): Motus unifies passive video generation, action-conditioned video prediction and action generation in a single 8B architecture. Evaluating the exact same fine-tuned Motus checkpoint across L1, L2 and L3 removes all cross-level model differences. Within identical weights, adjacent levels remain significantly correlated (L1 vs. L2: r = 0.359, p = 0.023; L2 vs. L3: r = 0.328, p = 0.039), whereas L1 vs. L3 is non-significant (r = 0.252, p = 0.116).
  • Consistency Across Model Families (Wan, Cosmos and FLUX): The findings from the same checkpoint Motus align with Wan (Wan2.2 -> Motus -> FastWAM: L1–L2 r = 0.359, p = 0.023; L2–L3 r = 0.332, p = 0.036) and Cosmos(Cosmos2.5 -> Action -> Policy: L1–L2 r = 0.343, p = 0.033; L2–L3 r = 0.378, p = 0.018) family pairings, where L1 vs. L3 remains non-significant (p > 0.47). To test a third independent family, we adapted FLUX-3-action across L1, L2 and L3, observing the same task-split degradation and adjacent-level association pattern (L1–L2 r = 0.439, p = 0.005; L2–L3 r = 0.315, p = 0.048 and L1–L3 r = 0.247, p = 0.125)

Supported vs. Unsupported Conclusions(Q2)

  • Conclusions Supported by the Evidence:

    • Adjacent-level association structure: Across an identical checkpoint (Motus), initialization-matched variants (Cosmos2.5, FLUX-3-action) and backbone-aligned pairings (Wan2.2), task-level scores consistently exhibit moderate positive associations between adjacent levels (L1–L2 and L2–L3), while passive prediction (L1) alone shows a weak, non-significant correlation with closed-loop real-robot progress (L3).
    • Skill-specific execution gaps: Contact-rich skills such as Pour , Scoop , Braid , and Plug achieve L2 scores of 0.50–0.68 but near-zero real-robot L3 success, even with the sameMotus checkpoint at both levels. This indicates that L2 prediction scores alone do not fully characterize closed-loop execution performance on these skills.
    • Non-equivalence of aggregate L1 and L3 model rankings: Stronger passive generation at L1 does not guarantee stronger closed-loop control at L3.
  • Clarifying Potential Misunderstandings in Phrasing:

    • As noted in Sections 5.1 and 6.2, our cross-system pairings (such as Wan2.2 -> Motus -> FastWAM) were intended to examine cross-level performance relationships within a shared model family rather than to attribute aggregate L3 score differences to video-backbone quality or claim causal capability transfer. Because shorthand phrases such as "capability transfer" and "backbone-matched" in Sections 1, 7.2, and 8 could invite that misreading, we will revise them to "cross-level performance associations" and "backbone-aligned families" for consistency with Sections 5.1, 6.2, and 7.1.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all three reviewers for recognizing Phys-WorldBench's unified three-level evaluation protocol within a shared physical interaction space and our real-robot dataset with physically executed counterfactual trajectories. We summarize our new experiments, statistical analyses, and planned revisions addressing the two shared themes across the reviews, followed by point-by-point responses to every Weakness and Question.

Cross-Level Relationships within a Single System and Across Model Families All three reviewers noted that comparing different systems across Levels 1–3 introduces architectural and training differences and asked whether the cross-level pattern holds under controlled setups and in a third model family. To address this, we conducted a controlled single-checkpoint evaluation using Motus and adapted a third independent model family (FLUX-3-action) :

Evaluation settingControl levelL1–L2 r (p)L2–L3 r (p)L1–L3 r (p)
MotusIdentical weights0.359 (0.023)0.328 (0.039)0.252 (0.116)
Cosmos FamilyShared initialization and training data0.343 (0.033)0.378 (0.018)0.118 (0.474)
Wan FamilyBackbone-aligned family0.359 (0.023)0.332 (0.036)0.099 (0.542)
FLUX-3-actionShared initialization and training data0.439 (0.005)0.315 (0.048)0.247 (0.125)
  • Single-Checkpoint Evaluation (Motus): we evaluate the same fine-tuned Motus checkpoint across L1, L2 and L3 to remove cross-level architectural and training differences. Within identical weights, adjacent levels remain significantly correlated (L1 vs. L2: r = 0.359, p = 0.023; L2 vs. L3: r = 0.328, p = 0.039), whereas L1 vs. L3 is non-significant (r = 0.252, p = 0.116).
  • Consistency Across Model Families (Wan, Cosmos and FLUX): The findings from the same checkpoint Motus align with Wan (Wan2.2 -> Motus -> FastWAM: L1–L2 r = 0.359, p = 0.023; L2–L3 r = 0.332, p = 0.036) and Cosmos(Cosmos2.5 -> Action -> Policy: L1–L2 r = 0.343, p = 0.033; L2–L3 r = 0.378, p = 0.018) family pairings, where L1 vs. L3 remains non-significant (p > 0.47). To test a third independent family, we adapted FLUX-3-action across L1, L2 and L3, observing the same task-split degradation and adjacent-level association pattern (L1–L2 r = 0.439, p = 0.005; L2–L3 r = 0.315, p = 0.048 and L1–L3 r = 0.247, p = 0.125)

Diagnostic Value and Robustness Checks for Level-2 Evaluation Reviewers also asked whether the composite Level-2 score (L2-PWS) provides true diagnostic value beyond task difficulty and single metrics, whether GT–CF comparisons are confounded by initial scenes and whether the metric remains valid with explicit object tracking and action-shuffling. Our additional analyses support the soundness of the Level-2 design on all three aspects:

  • L2-PWS adds predictive value beyond single metrics and observable task properties (tAc1 W1/Q1): Action-conditioned evaluation requires measuring both nominal demonstration accuracy (F) and counterfactual robustness (R). Empirically, across all three model families (Wan, Cosmos and FLUX-3-action), L2-PWS significantly improves L3 prediction over every single-clip metric (p_OLS <= 0.0010) and over R alone (p_OLS = 0.0353), remains statistically significant under every observable task-feature control (p_OLS <= 0.018, p_boot <= 0.020) as well as after controlling for the manipulation difficulty by the independent VLA pi_0.5 performance (p_boot = 0.007).

  • GT–CF performance shifts reflect counterfactual actions rather than initial-state differences (GypA W2/Q2): Across Motus, Cosmos2.5, and FLUX-3-action, prediction shifts between successful (GT) and failed (CF) runs are 1.35x–3.19x larger than normal trial-to-trial variation among successful runs (p <= 0.0172), remain virtually unchanged on subsets with closely matched initial frames (SSIM >= 0.80), and show no dependence on initial object placement offsets (p = 0.140)

  • Level-2 evaluations hold under pool-free normalization and object tracking, and genuinely depend on input actions (fWuV W1/Q1): Pool-independent Fixed-Anchor Normalization (rho = 0.95–0.97, preserving Motus > Cosmos-Pred. 2.5) and explicit manipulated-object tracking metrics (r = 0.96–0.98) both preserve task-level scores and significant L2–L3 correlations (p <= 0.031). Moreover, same-task cross-episode action shuffling significantly degrades gripper trajectory accuracy and PSNR in both Cosmos-Pred2.5 (p = 1.8e-15) and Motus (p = 1.3e-5 for accuracy, p = 1.8e-8 for PSNR), confirming active reliance on action inputs.

Taken together, these controlled evaluations and diagnostic ablations reinforce our benchmark design and empirical findings. In the revised manuscript, we will incorporate all new experiments and analyses and unify the terminology, scope statements, and metric descriptions throughout the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.