acceptodds
Under review as a conference paper at ICLR 2027

OneWorld: Learning Consistent Physics Across Actions in World Models

Abstract

Action-conditioned video world models aim to predict scene evolution under different actions, a capability that is essential for reliable planning, decision-making, and interaction in dynamic environments. However, futures generated independently from the same initial scene may each appear plausible while implying incompatible physical properties, such as friction or mass. This inconsistency can lead to contradictory predictions across interventions, making it difficult for the model to maintain a coherent understanding of the underlying world and limiting its reliability for planning and decision-making. To address these issues, we propose OneWorld, a shared-mechanism counterfactual generation framework that jointly models multiple action-conditioned futures under a common latent physical mechanism. A physical mechanism interpreter first infers a distribution over latent mechanisms from each action–outcome branch. These distributions are then aggregated into shared-world evidence, which captures whether the branches admit a common physical explanation while accounting for uncertainty in less informative branches. This evidence constrains flow training and guides sampling, encouraging consistency in the underlying physical mechanism while preserving the distinct outcomes induced by different actions. We further introduce a multi-intervention evaluation protocol in controlled environments, following the interaction settings of ACWM-Phys, to assess whether generated futures can be jointly explained by the same physical parameters, alongside standard measures of single-rollout prediction quality. Experiments in these environments show that OneWorld improves cross-intervention physical consistency while maintaining competitive single-rollout prediction quality.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
world models, video generation, counterfactual reasoning, action-conditioned generation, physical reasoning, 4D scene generation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_kqnoOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes OneWorld, a shared-mechanism counterfactual generation framework designed to address "shared-world inconsistency" in action-conditioned video world models. While independent rollouts under different action interventions can appear visually plausible, they may imply contradictory physical properties (e.g., conflicting friction or mass). To enforce consistency without forcing identical outcomes, OneWorld introduces a physical mechanism interpreter and a prior-corrected shared-world evidence score. This score provides relational supervision during training and trajectory-level guidance during sampling. The authors also introduce a simulator-based evaluation protocol to separate individual physical fit from cross-intervention shared-world consistency across rigid-body, deformable, and particle-based environments.

Critical strengths

The proposed simulator-based evaluation protocol makes a significant methodological contribution by cleanly separating single-rollout prediction quality (EindE_{ind}) from cross-intervention shared-world consistency (Δworld\Delta_{world}). This effectively addresses the long-standing evaluation dilemma in video world models, where traditional perceptual metrics (e.g., PSNR, SSIM, FVD) fail to detect physical violations masked by high visual realism.

Critical weaknesses

1. Fundamental Ill-Posedness Mischaracterized as a Model Flaw: The core premise of OneWorld relies on the assumption that alternative action-conditioned futures from a shared initial frame (I0I_0) must imply a single consistent physical world. However, predicting future videos from a single static observation and an action sequence is inherently a highly ill-posed inverse problem. Hidden physical properties—such as mass distribution, internal material stiffness, precise surface friction, and contact parameters—are fundamentally unobservable from a single RGB frame. In such information-under-specified settings, multiple futures with diverging physical manifestations are mathematically natural and legitimate outcomes. Characterizing this natural multi-valued uncertainty as "shared-world inconsistency" and penalizing it treats an inherent property of the problem formulation as a model defect.

2. Diversity Collapse and Overfitting to Discrete Training Priors: To force multi-branch trajectories to share a common physical explanation under severe information deficit, the proposed shared-world optimization (Lshared\mathcal{L}_{shared} and flow guidance) is likely to suppress generative diversity. Rather than learning true causal physical reasoning, the model is incentivized to collapse its predictions toward local mean trajectories that fit physical properties (distributions) seen during training. This mechanism risks causing a severe over-reliance on training priors, where the model merely memorizes and matches discrete parameter manifolds rather than genuinely generalizing to unconstrained physical dynamics.

Questions for authors

1. Continuous vs. Binary Treatment of Physical Discrepancy (y∈{0,1}y \in \{0, 1\}): The mechanism interpreter is trained using a binary cross-entropy loss over same-world (y=1y=1) and mismatched-world (y=0y=0) groups. However, the physical world is inherently continuous rather than strictly categorical. For instance, a mismatched pair with negligible parameter divergence (e.g., mass 1.0 vs. 0.99) is treated with the exact same binary penalty (y=0y=0) as a pair with extreme divergence (e.g., mass 1.0 vs. 0.01). Does treating physical distance as a binary indicator limit the interpreter's granularity? Would a continuous distance-weighted supervision or contrastive formulation be more principled to reflect varying degrees of physical incompatibility?

2. Prevention of Interpreter Collapse During Two-Stage Training: The shared-world optimization relies heavily on a frozen physical mechanism interpreter that was trained in the first stage. Since the video generator is subsequently optimized to maximize the shared-world evidence (Cτ\mathcal{C}_\tau) produced by this frozen interpreter, what prevents the video model from exploiting vulnerabilities or blind spots in the fixed interpreter (akin to adversarial exploitation), leading to representation collapse where the interpreter's outputs degenerate or lose physical meaning?

3. Generalization to Unseen Object Instances and Heterogeneous Properties: The paper evaluates generalization on interpolated parameters within known simulation spaces. However, real-world deployment often involves objects with entirely unseen physical properties (e.g., a cube with a radically different mass distribution, friction profile, or internal geometry compared to training instances). How does this framework handle out-of-distribution (OOD) object instances whose true physical parameters fall outside the support of the training distribution? Does the model risk forcing novel objects into familiar training "priors," thereby failing on genuinely new physical dynamics?

4. Consistency vs. Physical Validity (Hallucinated Consistency): Cross-intervention consistency ensures that multiple branches share a unified explanation, but it does not inherently guarantee physical correctness. Is it possible for the model to achieve high shared-world consistency (Δworld≈0\Delta_{world} \approx 0) by finding a "consistent" yet completely hallucinated or physically invalid explanation across all branches (e.g., uniformly violating momentum conservation or gravity across all rollouts)? How does the evaluation protocol or framework prevent models from optimizing for mere intra-group agreement at the expense of absolute physical reality?

2 comments
AuthorsOpenReview ↗

Response to kqno: Uncertainty, Diversity, and Physical Validity

We thank the reviewer. We distinguish uncertainty over the world from compatibility of interventions within one sampled world.

W1. Ill-posedness and the shared-world assumption

We agree that a single RGB frame does not identify hidden physics. Eq. (1) explicitly marginalizes this uncertainty:

p(V(1:K)∣I0,A(1:K))=∫p(W∣I0)∏kp(V(k)∣I0,A(k),W) dW.p(V^{(1:K)}\mid I_0,A^{(1:K)})=\int p(W\mid I_0)\prod_k p(V^{(k)}\mid I_0,A^{(k)},W)\,dW.

Slippery-world and rough-world groups can both be valid for one image. Within each group, actions intervene on one realization of W. Independent predictions can be valid marginal samples; they need not form a valid joint counterfactual sample. Our criticism concerns interpreting them as interventions on one world. We clarify this in Sec. 3.1.

The interpreter returns a distribution from each action/outcome pair. Shared evidence does not require identical videos/posteriors or one fixed mechanism across all groups. A posterior equal to the prior cancels exactly from the score (Appendix A.2). Neutrality does not prove calibrated learned uncertainty.

W2. Diversity and discrete-prior memorization

We retain flow supervision and test diversity on fixed Push Cube scenes/three-action sets, generating 20 groups with matched seeds. D is within-action mean pairwise terminal-position distance across repeats, normalized by the fixed position scale (Appendix E.13):

Method/referenceDShared errorGap
ACWM-DiT0.08740.099260.02371
Posterior Alignment0.04730.080960.00762
OneWorld0.08460.078320.00386
Simulator reference0.08610.002290.00038

The simulator reference samples hidden configurations from the empirical training distribution with fixed visible states/actions. OneWorld retains spread with better fit/consistency, arguing against terminal-position collapse here without proving full distributional coverage. Separately, holding out both physical factors jointly reduces mean normalized parameter error (NPE) from 0.226 to 0.082; the evaluator includes the unseen values rather than snapping to the training grid (Appendix E.9).

Q1. Binary labels versus continuous discrepancy

Binary labels supervise shared origin. Posteriors and scores remain continuous; equal negative labels need not yield equal BCE losses. However, BCE does not enforce a calibrated or monotonic relation to parameter distance. Such distance is also action-dependent in its observable consequences: friction differences may be invisible without sliding.

We compare binary BCE with soft targets 1-r and negative weights 1+r, where r is the RMS pairwise parameter difference after training-range normalization. Under matched groups, architecture, budgets, and three seeds, Push Cube interpolation results are:

SupervisionAUROCGapNPE
Binary BCE0.95930.004380.0867
Soft-label BCE0.93520.007910.1129
Distance-weighted BCE0.94760.005620.0984

These alternatives do not improve results (Appendix E.9); better continuous objectives remain possible.

Q2. Exploitation of the frozen interpreter

Freezing prevents the criterion from co-adapting with the generator; it does not guarantee immunity to exploitation. Retained action-conditioned flow supervision constrains the generator. Crucially, evaluation uses separate frozen state extractors, simulator replay, and parameter fitting, with no interpreter scores/features (Appendix E.11). Lower independent fitting errors and higher parameter recovery (Q4) provide evidence beyond the optimized score. This does not formally exclude adversarial blind spots.

Q3. OOD objects and physics

Without retraining, we extend Push Cube drag above its training maximum by 10% and 25% of the training-range width. NPE improves from 0.272 to 0.164 and from 0.400 to 0.272; shared error improves from 0.11586 to 0.08553 and from 0.13939 to 0.10460. The common evaluator grid includes these OOD values (Appendix E.10). Errors grow at the larger extension. This tests parameter extrapolation within the same object family, not unseen internal geometry, mass distributions, or arbitrary friction profiles. Bias toward familiar dynamics remains a risk outside the tested support.

Q4. Consistency versus physical validity

Yes: a small gap alone can reflect consistently wrong predictions. We therefore jointly report individual/shared simulator fit and recovery of the true generating parameters. On Push Cube, individual error falls from 0.07535 to 0.07384, shared error from 0.09788 to 0.07741, and recovery rises from 59.4% to 87.5%; mean recovery across environments rises from 57.3% to 85.0% (Appendix E.8). These assess simulator fit and agreement with the actual world beyond intra-group agreement, within the measured states/simulator family. They do not certify every conservation law.

Reviewer_kqnoOpenReview ↗

Reponse to rebuttal

Thank you for your explanation. My major concern remains: does the consistency matter? Will a consistent world be more physically plausible? Current experiments and metrics do not prove this. I also don't see convincing real-world applications. Thus I maintain my score as 2.

Reviewer_JH7dOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

OneWorld targets a failure mode per-rollout metrics miss: futures predicted for different actions from the same scene may each look fine while jointly contradicting each other physically. A mechanism interpreter scores whether the branches share one physical explanation; trained contrastively and frozen, it shapes both training and sampling. In three small environments, an independent simulator-replay evaluation shows the consistency gap falling 84.3% versus ACWM-DiT, with physics-configuration recovery up from 57.3% to 85.0%.

Critical strengths
  1. The paper makes "futures of different actions should belong to the same world" precise and measurable. For planning this is an obvious requirement: comparing candidate actions under mutually inconsistent physics is meaningless. As far as I know, the formalization and the independent simulator-based evaluation are new, and they will outlive this particular method.
  2. The method fits the question and the analysis is unusually complete: prior correction avoids double counting, uninformative branches drop out exactly, and beyond the 84.3% number there are recovery, joint interpolation, grid density and component ablations.
Critical weaknesses
  1. All quantitative evidence comes from small controlled environments: Pymunk for the two pushing tasks, a particle simulator for pouring, two physical factors each on 3×3 or 4×4 grids, one backbone, three branches by default. RoboDesk is qualitative only, and the BAIR/RoboNet appendix tests the backbone without any OneWorld components, so nothing here speaks to complex scenes.
  2. The motivation is planning, yet the paper never shows that better consistency leads to better action choices. The guidance is also not free: about 64% more latency and 3.3× peak memory per group, which matters for the MPC-style use cases the paper itself invokes.
Questions for authors
  1. If candidate action sequences are ranked by predicted outcomes, does OneWorld actually pick better plans than ACWM-DiT?
  2. In Fig. 4a, why does Δworld\Delta_{\mathrm{world}} grow with KK? Estimation variance, optimization, or something intrinsic? Any numbers at K=10K = 10 or more?
  3. The interpreter is trained per environment on simulator-generated groups. Could one interpreter be shared across environments, or reused for new physics without retraining?
2 comments
AuthorsOpenReview ↗

Response to JH7d: Evaluation, Planning, Scaling, and Interpreter Reuse

We thank the reviewer for recognizing the formulation and independent evaluation. We address the evaluation scope, downstream benefit/cost, branch scaling, and interpreter reuse below.

W1. Beyond the small controlled environments

We agree that the original BAIR/RoboNet results evaluate the backbone and do not establish OneWorld's physical consistency in complex scenes. We extend RoboDesk with quantitative prediction and action selection on Open Drawer and Sliding Door (Sec. 4.4; Appendix C.4). Each task uses 240 goal/action-set pairs across 48 held-out states, three candidates, and three generation repeats. Methods share inputs and a frozen state extractor. Errors are normalized by joint travel; success requires actual goal error at most 0.10.

TaskMethodState errorGoal errorSuccess (%)
Open DrawerACWM-DiT0.10840.164037.36
OneWorld0.07460.150840.28
Sliding DoorACWM-DiT0.12820.169136.25
OneWorld0.08960.143442.92

Paired 95% scene-bootstrap intervals for goal-error differences (OneWorld minus ACWM-DiT) are [-0.0193, -0.0071] and [-0.0335, -0.0180]. Open Drawer's success-gain interval includes zero; Sliding Door's is [3.47, 9.86] percentage points. This extends evaluation to robot contact and joint-constrained motion, but remains controlled simulation with one backbone, rather than evidence of unrestricted real-world generalization.

W2/Q1. Better action choices and their cost

Yes, in our added open-loop Push Cube test. Both methods rank the same three candidate actions by predicted endpoint distance to a goal; the selected action is then executed in the original simulator world. We use 320 tasks across 32 held-out states and three repeats, with no access to hidden physics or actual candidate outcomes during selection. Success means actual normalized goal distance at most 0.15; regret is excess actual distance over the best candidate (Sec. 4.3; Appendix E.12).

MethodSuccess (%)Goal distanceRegrets/group
ACWM-DiT44.380.20430.081213.247
OneWorld without guidance50.310.18060.057513.245
OneWorld full60.630.14820.025021.684
Simulator oracle71.880.12310--

Full OneWorld gains 16.25 success points and reduces regret by 69.2%. Without guidance, the same trained checkpoint improves selection at baseline inference cost. Full guidance adds 10.32 success points over this variant; the paired 95% scene-bootstrap interval is [7.08, 13.44]. These results support downstream utility, without identifying consistency as the sole cause of the gain.

The cost concern is valid: full guidance uses 1.637 times the latency and 3.27 times the peak memory (7.864 vs. 2.403 GiB/GPU). Measurements use eight A100 GPUs, FP32, 50 flow steps, and three 37-frame branches at 240 x 240; they exclude extraction and selection (Appendix D.7). We claim open-loop action-selection gains, not real-time or closed-loop MPC capability.

Q2. Why does the gap increase with branch count?

More interventions can expose incompatible physics that fewer branches leave undetected. This is not an unavoidable scaling law: fitting errors average over branches, so the gap does not mechanically grow as a sum. We extend the Push Cube test to K=6,8K=6,8, fixing checkpoints and guidance settings, and compare evidence-sampling budgets M=64,256M=64,256 (Appendix D.6):

KACWM-DiT gapOneWorld M=64OneWorld M=256
30.022530.003570.00341
50.025870.004160.00394
60.027420.004830.00497
80.031960.006280.00573

Increasing M changes the gap modestly and does not remove the trend; even ACWM-DiT, which does not use the evidence estimator, shows growth. This argues against estimator variance as the sole explanation. The experiment does not disentangle optimization effects from the additional compatibility constraints, so we avoid attributing the trend to a single cause.

Q3. Interpreter reuse

Per-environment training remains our default. In an added reuse test, we freeze the interpreter and prior trained on one RoboDesk task and evaluate them on the other, without target-task training or threshold adjustment. Drawer-to-Door transfer achieves 76.04% accuracy / 0.8407 AUROC; Door-to-Drawer achieves 80.21% / 0.8806. Target-trained accuracies are 90.63% and 92.71%, respectively (Appendix C.3).

This supports partial reuse between related tasks, with degradation. It tests compatibility classification, not generation with a transferred interpreter or a universal interpreter across dynamics families. Joint interpolation supports reuse on unseen in-range physics; arbitrary new physics remains unestablished.

Reviewer_JH7dOpenReview ↗

Thanks the authors for response

The new physical-validity analyses and closed-loop planning experiments are valuable additions, but the evidence is still limited to controlled simulation, so I will maintain my weak accept rating.

Reviewer_Vd6XOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper is well motivated by the observation that different action-conditioned futures generated by existing world models from the same initial scene may imply inconsistent underlying physical properties. Such inconsistency could negatively affect downstream tasks like planning and decision-making. The authors propose OneWorld, which explicitly encourages different action-conditioned futures to share consistent physical mechanisms. Experiments show that the proposed method can generate videos with more consistent underlying physical properties across different actions.

Critical strengths
  1. The motivation is clear. The paper identifies an important limitation of existing action-conditioned world models: different action-conditioned futures may be individually plausible but inconsistent in their underlying physical properties.

  2. Method design is well-motivated. The proposed method directly addresses the identified challenges. In particular, the shared-world evidence accounts for uncertainty across different action branches, rather than simply enforcing identical physical representations.

Critical weaknesses
  1. Lack of metric interpretability and reference. It is difficult for readers to judge whether the improvements reported in Table 1 are practically significant, as the absolute differences between methods are relatively small. What is the scale of these metrics, and how should we interpret the reported values? I think a ground-truth simulator rollout would be an important reference baseline. Since the authors have access to the simulator and ground-truth physical parameters, they could directly evaluate ground-truth simulator videos using the same evaluation pipeline. Without a reference, it's hard to tell whether the proposed method can achieve significant improvement or not.

  2. Lack of downstream evaluation. The authors explicitly emphasize the importance of physical consistency for planning and decision-making in the Introduction. However, physical consistency is only a reasonable prerequisite for reliable planning, and improving physical consistency does not necessarily lead to better planning performance. I think adding at least one downstream planning task would make this work more solid and better demonstrate the importance of consistent physics.

Questions for authors

Please see the weaknesses part.

3 comments
AuthorsOpenReview ↗

Response to Vd6X: Metric Interpretation and Downstream Evaluation

We thank the reviewer for recognizing the motivation and uncertainty-aware design. We address both concerns with a GT-video reference and a downstream action-selection evaluation.

W1. Metric scale and practical significance

EindE_{\mathrm{ind}} fits each predicted branch with its own simulator parameters; EsharedE_{\mathrm{shared}} fits all branches with one parameter configuration. Their difference, Δworld=Eshared−Eind\Delta_{\mathrm{world}}=E_{\mathrm{shared}}-E_{\mathrm{ind}}, measures the additional cost of requiring shared physics. These are dimensionless, training-normalized trajectory costs, not percentages; they are not bounded by one. Their scales differ across environments, so comparisons should be made within each environment. A small gap alone does not imply accurate physics; it must be read alongside absolute fitting error and parameter recovery.

Following your suggestion, we evaluate GT simulator videos through the same preprocessing, frozen state extractors, candidate grids, and fitting pipeline (32/32/40 held-out groups for Cube/Rope/Water; three actions per group):

EnvironmentMethodIndividual errorShared errorGap
Push CubeACWM-DiT0.075350.097880.02253
OneWorld0.073840.077410.00357
GT videos0.001830.002170.00034
Push RopeACWM-DiT0.041280.055170.01389
OneWorld0.040120.042210.00209
GT videos0.003460.003920.00046
Pour WaterACWM-DiT0.090370.125410.03504
OneWorld0.088460.094020.00556
GT videos0.009270.010310.00104

OneWorld closes 85.44%, 87.86%, and 86.71% of the ACWM-DiT-to-GT gap, using (ΔACWM−ΔOW)/(ΔACWM−ΔGT)(\Delta_{\mathrm{ACWM}}-\Delta_{\mathrm{OW}})/(\Delta_{\mathrm{ACWM}}-\Delta_{\mathrm{GT}}). Shared fitting error decreases by 20.9%, 23.5%, and 25.0%; mean exact physical-configuration recovery increases from 57.3% to 85.0%. The GT results are an empirical reference, not a theoretical lower bound: extraction and fitting introduce residual error even for simulator videos. OneWorld still has substantially higher absolute fitting error than GT, so we do not claim simulator-level accuracy. The revised manuscript includes metric definitions in Sec. 4.1/Appendix E.4 and GT references in Appendix E.8.

W2. Does consistency help downstream decisions?

We agree that consistency alone does not establish planning performance. We therefore evaluate open-loop Push Cube target reaching (Sec. 4.3; Appendix E.12). Each model receives identical initial scenes, goals, and three candidate action sequences. A common frozen extractor reads predicted endpoints, and we execute the candidate predicted to end closest to the goal. Hidden physics and actual candidate outcomes are unavailable during selection. Success means actual normalized goal distance at most 0.15; regret is excess actual distance over the best candidate.

We use 320 tasks across 32 held-out states, with three generation repeats and no best-of-repeat selection:

MethodSuccess (%)Goal distanceRegret
Random selection25.830.28500.1619
ACWM-DiT44.380.20430.0812
Posterior Alignment52.080.17590.0527
OneWorld without guidance50.310.18060.0575
OneWorld full60.630.14820.0250
Simulator oracle71.880.12310

Full OneWorld improves success by 16.25 percentage points and reduces regret by 69.2% relative to ACWM-DiT. The respective 95% scene-bootstrap success intervals are [55.63, 65.52] and [40.00, 48.85]. The oracle uses the same candidates, so its success need not be 100%. Without guidance, OneWorld also improves action selection at essentially baseline generation cost (13.245 vs. 13.247 s/group).

These results provide direct evidence of better candidate-action selection in the tested setting. They do not establish closed-loop MPC performance or isolate consistency as the sole cause, since prediction quality also changes. We have made this scope explicit in the revised manuscript.

Thanks for the response. My first concern was largely addressed by the ground truth simulator reference results. However, my second concern was only partially addressed. I am not fully convinced by this provided simple open-loop action-selection task. I think more challenging planning tasks involving diverse scenarios and complex physical interactions would be necessary to better demonstrate the practical benefits of consistent physics. In particular, multi-step planning or closed-loop decision-making would provide stronger evidence of the proposed method's effectiveness. The original ACWM-Phys benchmark covers eight environments across four physical regimes, but the downstream evaluation is limited to Push Cube alone. Similarly, the main experiments are limited to only three environments covering three physical regimes, without sufficient justification for this limited selection. Overall, I appreciate authors efforts, but I will maintain my current rating.

AuthorsOpenReview ↗

Follow-up: Closed-loop Planning and Broader Physical Coverage

Thank you for the follow-up. We have completed closed-loop planning experiments on Push Rope and two RoboDesk tasks, and added Reacher to the controlled physical-consistency evaluation.

1. Closed-loop planning across three tasks

We evaluate rope-shape matching, drawer opening, and sliding-door positioning. At each decision, both methods predict three candidate sequences, each with three 12-step action stages. A common frozen extractor ranks their predicted final shapes or positions against the goal. We execute the first six steps at 10 Hz, obtain a fresh observation, and replan, for up to eight decisions per episode. Initial poses, goals, and stiffness/damping or friction/damping settings vary across episodes; hidden physics stays fixed within each episode.

Methods share training clips, candidate-generation rules, control limits, prediction horizons, and scoring functions. Hidden parameters and actual candidate outcomes are unavailable to either planner. Push Rope uses 32 held-out states; each RoboDesk task uses 48. Five reachable goals per state and three repeats yield 480/720/720 episodes per method. Goal-generating actions are withheld from the planner.

Rope error is symmetric mean nearest-point centerline distance normalized by the workspace diagonal; drawer/door error is joint-position error normalized by joint travel. Success requires five consecutive control steps within thresholds of 0.030.03 and 0.100.10. All episodes contribute to final error and decision counts; failures use all eight decisions.

TaskMethodSuccess (%) ↑\uparrowGoal error ↓\downarrowDecisionss/decision
Push RopeACWM-DiT44.580.05076.57313.812
Push RopeOneWorld59.790.04206.05022.623
Open DrawerACWM-DiT47.080.15066.52514.065
Open DrawerOneWorld58.470.12716.25422.952
Sliding DoorACWM-DiT48.890.15936.53213.952
Sliding DoorOneWorld64.170.12546.06222.768

Both methods use the same hardware, FP32, and 50 flow steps. Latency includes generation, extraction, and scoring. We average goals and repeats within each state, weight states equally, and use 4,000 paired whole-state bootstrap resamples, retaining all goals and repeats together. Differences below are OneWorld minus ACWM-DiT, with 95% confidence intervals:

TaskSuccess difference (percentage points)Goal-error difference
Push Rope15.21 [10.62, 19.79]-0.0087 [-0.0124, -0.0052]
Open Drawer11.39 [8.33, 14.44]-0.0235 [-0.0302, -0.0168]
Sliding Door15.28 [11.39, 19.31]-0.0339 [-0.0444, -0.0243]

OneWorld improves success by 11.39–15.28 percentage points and reduces final error by 15.6–21.3%. All paired intervals favor OneWorld. It also uses fewer decisions, with per-decision latency increasing by approximately 1.631.63–1.64×1.64\times. These results show improved task performance under repeated interaction and feedback across deformable-object and articulated-object tasks.

2. Broader physical coverage: Reacher

The original Push Cube, Push Rope, and Pour Water environments support controlled interventions with known physics across rigid-body, deformable, and fluid dynamics. We add Reacher, a two-link torque-controlled arm, to cover kinematics. Each link independently uses mass/inertia multipliers of {0.75,1.00,1.25}\{0.75,1.00,1.25\}, giving nine configurations at fixed geometry and appearance.

We use 128/32/32 disjoint training/validation/test states crossed with all nine configurations. Each group has three actions; three generation repeats yield 864 test groups per method. Both methods share the backbone, data, initialization, and 12,000 generator updates; OneWorld uses 20,000 interpreter updates.

A frozen extractor recovers elbow/tip trajectories. Fitting uses mean trajectory discrepancy normalized by arm length, with the same definitions as the paper:

E_{\mathrm{shared}}=\min_w\frac{1}{K}\sum_k\ell_k(w),\qquad \Delta_{\mathrm{world}}=E_{\mathrm{shared}}-E_{\mathrm{ind}}.$$ | Method/reference | M-MSE | $E_{\mathrm{ind}}$ | $E_{\mathrm{shared}}$ | $\Delta_{\mathrm{world}}$ | Recovery (%) $\uparrow$ | |---|---:|---:|---:|---:|---:| | ACWM-DiT | 0.02867 | 0.06183 | 0.07946 | 0.01763 | 64.93 | | OneWorld | 0.02812 | 0.06091 | 0.06408 | 0.00317 | 86.92 | | GT videos | n/a | 0.00241 | 0.00276 | 0.00035 | 95.49 | OneWorld reduces the gap by **82.0%** and shared fitting error by **19.4%**, while recovery rises by **21.99 percentage points**. GT videos use the same extraction and fitting pipeline. Together, the additions extend physical coverage to four categories and demonstrate closed-loop planning gains on three tasks. The revised manuscript reports these results in Sections 4.2–4.3 and Appendix E.13.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.