acceptodds
Under review as a conference paper at ICLR 2027

From Pixels to Executable Physics: Behavior-Grounded World Models for Embodied Action Consequence Prediction

Abstract

In outdoor environments, varying terrain and contact conditions can cause identical action commands to produce different motion outcomes for quadruped robots. Predicting these outcomes is essential for reliable navigation and informed path selection. Existing video world models can generate visually plausible futures, yet often struggle to capture robot dynamics and terrain interactions, leading to overly optimistic predictions of action outcomes. We present DRWM, a physically consistent world model that incorporates robot–terrain contact dynamics to predict the consequences of quadruped actions. Starting from an initial RGB image, our method reconstructs scene geometry and integrates dense, dimensionless priors over terrain contact properties, calibrated using real-world action–response data. The resulting terrain model supports physics simulation with spatially varying contact properties such as friction. Given the robot’s initial state and candidate action sequences, we simulate its future motion and use the resulting camera trajectories and rendered geometry to guide the training of a video world model. Generated future observations are then used to update the terrain reconstruction, enabling iterative model rollouts without additional real-world images. Extensive experiments demonstrate that DRWM substantially outperforms geometry-only baselines and baselines with uniform physical parameters on key physical metrics characterizing action outcomes, while achieving state-of-the-art performance on all reported video evaluation metrics. These physically grounded predictions support more informed action selection: across 90 challenging scenarios, an 8B vision–language model augmented with DRWM outperforms a 32B vision–language model on five of six evaluation metrics.

Primary area:
applications to robotics, autonomy, planning
Keywords:
World Model, Physics;Behavior-Grounded;Robot Navigation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_jxAiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DR-WM for predicting robot motion and future observations in outdoor quadruped navigation. It combines terrain-property prediction, physics simulation calibrated against real robot responses, and video generation. The predicted videos are also used to help a VLM choose actions. Experiments evaluate video quality, motion prediction, and action selection.

Critical strengths
  1. The motivation is clear. Different terrain conditions can lead to different outcomes for the same robot command, making this a relevant prediction problem.
  2. The physical model is calibrated against real robot responses. Tables 4 and 13 show improved motion prediction over the default and uniform-parameter settings.
  3. The supplementary experiments are useful. They include comparisons with lookup-based methods, video-generation ablations, and multi-step predictions without future real observations (Appendices C, E, and H).
Critical weaknesses
  1. The contribution of PhyMap is not fully established across the evaluation stages. Table 9 compares physical-prior prediction, Tables 4 and 13 evaluate motion prediction, and Table 5 evaluates action selection. However, the paper does not clearly connect these results or show that the advantage in physical-prior prediction leads to the downstream gains. The training and adaptation protocols of the Table 9 baselines also need clarification.
  2. The physical-property evaluation measures agreement with proxy targets. Since the targets are derived from semantic and language-model priors, Table 9 can compare prediction agreement, but it does not independently validate the absolute correctness of the physical properties. The paper should distinguish proxy-target accuracy from recovery of real terrain properties.
  3. The evaluation protocol in Table 5 limits the interpretation of the action-selection result. Section 4.4 describes candidate-wise rollouts and reports metrics for the VLM-selected action, but does not clearly explain how candidate actions and the selected-action outcome are obtained. This makes it difficult to assess what the reported gain represents.
Questions for authors
  1. For Table 9, did the compared methods use the same physical-prior targets, data, test split, and adaptation protocol as PhyMap? Were they evaluated zero-shot or trained/fine-tuned on Physcene?
  2. How do the improvements in physical-prior prediction and motion prediction translate into action-selection performance?
  3. How are candidate actions generated, and how is the outcome of the VLM-selected action in Table 5 obtained?
  4. What independent evidence supports the physical-prior values beyond agreement with the lookup-derived targets?
4 comments
AuthorsOpenReview ↗

Response to Question 1

We sincerely appreciate the time and care you devoted to reviewing our work and providing constructive feedback. All VLM-based baselines were evaluated in a zero-shot setting on Physcene, without additional fine-tuning. To maintain consistent physical prior definitions, their prompts included the same 32-class mapping to friction, roughness, and stiffness used to construct PhyMap’s dense supervision. Accordingly, PhyMap learned physical prior from supervised training on Physcene. These evaluation settings are clarified in the revised Appendix C.

AuthorsOpenReview ↗

Response to Question 2

Your detailed feedback is greatly appreciated. PhyMap estimates terrain contact properties, which Genesis uses to predict how the robot will move under each candidate action. The resulting future frames help the VLM compare progress, drift, and stability across candidates. Tables 4 and 13 show the gains in motion prediction. Table 5 shows that, with the full DR-WM rollouts, the VLM selects actions that make more progress with less tilt and slip in simulation. We explain this connection in the newly added Appendix I.

AuthorsOpenReview ↗

Response to Question 3

Thank you for the time and effort. A VLM proposes candidate maneuvers based on the current image and navigation goal. Each candidate is paired with a locomotion command sequence, and all settings use the same candidates. The selection VLM returns the ID of its chosen candidate. For Table 5, we compute the metrics from that candidate’s simulation logs, recorded during execution by the official locomotion policy. The newly added Appendix I describes this procedure and provides example prompts.

AuthorsOpenReview ↗

Response to Question 4

We are grateful for your careful reading and valuable comments. Tables 4 and 13 provide evidence independent of the lookup table labels. Both experiments compare simulated motion with real robot motion from test scenes held out from calibration. In Table 4, Full PhyMap has lower motion prediction errors than Uniform PhyMap. Both use the same scene geometry, control commands, and mean contact parameter values. Table 13 shows similar gains over longer rollouts.

DR-WM uses physical priors inferred from images to predict how a given robot will interact with the terrain. For example, the coefficient of friction depends on the foot material, the terrain surface, and the contact conditions. This makes it very difficult to estimate actual contact parameter values from images alone. We use real robot action and response data to calibrate these priors and obtain effective contact parameters in Genesis. We then evaluate the calibrated priors in motion prediction experiments.

Reviewer_8MQcOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

DR-WM is a framework for predicting terrain-dependent action consequences for quadruped robots. Given an initial RGB image, robot state, and candidate action commands, it (i) reconstructs scene geometry and predicts dense normalized physical priors (friction, roughness, stiffness) through a trainable head (PhyMap) on a frozen 3D-reconstruction backbone; (ii) maps these priors to executable contact parameters via a monotone mapping calibrated against real robot action-response data (behavior grounding); (iii) simulates future robot motion deterministically with the Genesis physics engine; (iv) conditions a Wan-based video diffusion model on the resulting camera trajectory and rendered geometry to generate future frames; and (v) feeds generated frames back into map reconstruction for iterative long-horizon rollout. It introduces Physcene (2.58M images with physical priors; 100K+ robot clips). Experiments compare against action-conditioned video world models, ablate the executable prior, and evaluate VLM-based action selection over 90 scenarios.

Critical strengths
  1. PhyMap is a genuine, if incremental, contribution. Dense, feed-forward per-pixel physical-property prediction aligned to 3D geometry is far faster (6.2 vs 0.057 FPS) and scene-wide compared to multi-stage VLM/GS baselines (SiPhy, PhysGS, PUGS).
  2. Substantial dataset. Physcene's synchronized RGB/LiDAR/state/command collection with physical-prior labels across diverse terrain would be valuable to the community, conditional on release.
Critical weaknesses
  1. The "world model" framing overstates the contribution. The dynamics p(s|s,a) are a deterministic Genesis engine, not a learned model; the learned parts are a perception head, a monotone calibration map, and a video renderer. The core is a behavior-calibrated simulator plus a renderer, not a learned predictive world model in the Dreamer/Genie sense. The paper should state plainly what is learned vs. hand-engineered.
  2. The action-selection result (Table 5) is likely circular and has no real-world validation. The progress/tilt/slip/arrival metrics are measured on the DR-WM closed-loop rollouts themselves (their source is never stated), so the VLM is evaluated against the same simulator that produced its inputs. No real-robot experiment shows DR-WM improves actual decisions — the largest gap between the "practically useful for real-world navigation" claim and the evidence.
  3. Unfair video comparison. DR-WM's video is conditioned on the simulated camera trajectory and rendered geometry, while baselines receive only actions/prompts (or measured poses for camera-controlled ones). "Beats baselines on action adherence / physical realism" largely reflects having a simulator the baselines lack, not a stronger video model; a fair ablation would give baselines the same simulated trajectory.
  4. Confounded headline. "8B+DR-WM beats 32B alone" compares a model with future rollouts against one without; the clean comparison is same-size (8B vs 8B, 32B vs 32B). "5/6 metrics" counts a negligible arrival-time gain (0.0105 s) and hides that DR-WM is worse on path efficiency; Generic WM (Wan) worsens 8B's tilt/slip, showing video value is conditional on physics accuracy.
Questions for authors
  1. Are the Table 5 progress/tilt/slip/arrival metrics measured on the same Genesis rollouts DR-WM generates for the VLM? If so, is the evaluation circular, and is there any real-robot evidence that decision-making improves?
  2. Where is the learned dynamics? Genesis is deterministic, so what distinguishes this from standard real-to-sim identification — and is it still a "world model" without the video renderer?
  3. Have you conditioned baselines on the same simulated trajectory to isolate video-model quality from the "having a simulator" advantage?
  4. Since generic video hurts, have you tried giving the VLM the simulated state/metrics (tilt, slip, arrival) directly without video? What does video add over the numbers?
3 comments

Thank you for conducting the additional ablation study and providing further clarification. I have two remaining questions:

1. Fidelity of the evaluation scenes. How are the geometry and contact parameters of the independently constructed Genesis evaluation scenes obtained, respectively, and how is their consistency with the corresponding real-world terrain ensured? Have the simulated dynamics been quantitatively validated against real-robot executions on the same terrain under matched actions? Without such validation, it remains unclear how reliably the simulation metrics reflect real-world performance.

2. Effect of additional motion metrics. For Qwen3-VL-32B, providing both DR-WM videos and motion metrics consistently performs worse on all six evaluation metrics than providing videos alone. Could the authors explain this result? In particular, the generated videos are conditioned on Genesis-predicted robot poses and camera trajectories, while the four motion metrics summarize the predicted motion from the same rollouts. If these underlying motion predictions or metrics are inaccurate, why does the video-only condition still achieve the best performance?

AuthorsOpenReview ↗

Response to Question 1

Response to Question 1

Thank you for asking us to clarify how we built and validated the Genesis evaluation scenes used in Table 5.

We collected terrain data with a DEEP Robotics Lynx robot equipped with four RGB cameras and two LiDAR sensors, one at the front and one at the rear. The robot traversed each real scene several times and recorded synchronized RGB images and LiDAR scans. These passes provided denser terrain measurements from different viewpoints. We aligned the LiDAR scans in a common coordinate system and merged them. We then used the merged point cloud to build the terrain collision geometry for scoring in Genesis.

We calibrated the effective contact parameters using the procedure in Appendix B.2. We fitted these parameters to reduce differences in trajectory, velocity, and orientation between simulated and real robot motion under the recorded commands. We built the scoring scenes independently of the terrain maps that DR-WM reconstructs when predicting candidate outcomes. For each test case, all methods used the same scoring scene, contact parameters, initial robot state, and low-level controller.

Table 13 originally reported motion prediction errors on 50 real-world scenes. To evaluate the fidelity of the Table 5 scoring scenes directly, we extracted the 90 sub-scenes used in the VLM action-selection experiment from those 50 scenes. We replayed the recorded control commands in the corresponding Genesis scoring scenes and compared the simulated motion with the real robot's motion under the same commands. The last row below reports the results of this evaluation.

Genesis configurationATE @ 4.8 s (m) ↓ATE @ 9.6 s (m) ↓RotErr @ 9.6 s (°) ↓Velocity RMSE @ 9.6 s (m/s) ↓
Frozen Initial Map0.6171.34516.3070.534
Online Geo. + Frozen Physics0.6081.33413.7730.425
Online Geometry Only0.6531.40415.7810.392
Online Uniform PhyMap0.6041.26714.1980.359
ActiveSAM0.6121.34114.1260.387
Full DR-WM0.5681.24213.0960.322
Full DR-WM*0.5711.23813.1120.321

* This row reports the replay of recorded commands in the Genesis scoring scenes for the 90 VLM action-selection sub-scenes in Table 5.

AuthorsOpenReview ↗

Response to Question 2

Response to Question 2

Thank you for the question. We examined several cases in which Qwen3-VL-32B changed its choice after receiving the motion metrics. With video alone, the model chose routes that kept the robot farther from obstacles or path edges. When we added the motion metrics, it switched to routes that scored better on some predicted motion metrics but passed closer to obstacles or path edges in the videos. In separate Genesis evaluation rollouts, these newly selected actions led to noticeable slipping or collisions.

The videos and the four motion metrics are based on the same Genesis rollouts. Even when the four metrics accurately summarize the predicted motion, they leave out how close the robot comes to obstacles or path edges along the route. The videos show these spatial relationships. Even with some trajectory error, the videos can still convey which route gives the robot more clearance. These cues help explain why the model made better choices with video alone.

We also invite the reviewer to visit our anonymous demo website. It includes examples from several navigation scenarios that show how visual cues, such as obstacle clearance and path boundaries, can help the model choose safer routes. DR-WM Demo

Reviewer_QJojOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Given a single input RGB image, the initial state of a quadruped robot, and a planned robot action sequence, the paper uses physics simulation to estimate the camera trajectory that the robot would go through if it executed this action in the world, then uses that trajectory and the accumulated 3D point cloud to condition a video generative model. The set of video frames up to now is first 3D-reconstructed with frozen MapAnything, which is augmented with dense 2d heads for physical intrinsics such as friction, roughness, and stiffness. The ground truth for these physical heads is obtained by running a segmentation model then assigning each semantic class to LLM-derived physical property values. Then, Genesis simulator is used with the robot, the planned robot action sequence, the 3D terrain, and these estimated terrain physical properties to generate the camera trajectory that the robot would go through if it executed this action. This camera trajectory and the 3D terrain is used to condition a Wan video generative model (similar to Spatia) to output the video that the robot would see if it executed this robot action sequence.

Critical strengths

The overall task and pipeline of conditioning a video generative model on camera trajectories outputted by physical simulation is interesting. Normally camera trajectories are a conditioning input, and converting quadruped robot locomotion actions into camera trajectories can be more effectively modeled by using physics simulator in the loop.

The results are better than baselines in Table 1 and 2 as well as the user study in Table 3.

There is a substantial real-robot data collection effort to obtain paired robot action sequences and resulting observed videos validate the pipeline.

Critical weaknesses

Obtaining dense 3D ground-truth physical properties in real-world scenes is very challenging, and the method used by the paper is to train on pseudo-GT physical property mappings generated by segmenting the scene into a fixed set of 32 categories, then using a fixed LLM-generated lookup table to map those semantic classes to physical property values. Why do we even need to train dense physical property estimation heads on top of MapAnything? At test time it could be possible to directly run ActiveSAM and use the same lookup table to generate the physical property maps, following the pipeline used to create the GT - and since this is the ground-truth for the MapAnything pipeline it would theoretically be better despite requiring a separate model call. Table 4 and Table 13 could benefit from including this online ActiveSAM baseline.

Table 9 is misleading as it compares different categories of results. ActiveSAM's speed if run additionally at runtime should be provided as the true comparison to the method's 6.231 fps.

Spatia seems like the closest baseline for the video generative model part of the work, and provides the 3d control mechanism, but does not seem to be included in any baseline tables (e.g. Table 1).

The physical property maps estimated by the fine-tuned MapAnything look blurry in Figure 6 and Figure 7. Many small patches on the ground seem to have inaccuate shape in the prediction.

Questions for authors

As above.

4 comments
AuthorsOpenReview ↗

Response to Question 1

We sincerely appreciate your thoughtful feedback and constructive suggestion. We have added an online ActiveSAM baseline to Tables 4 and 13. This baseline uses MapAnything to reconstruct geometry. It then runs ActiveSAM online and uses the same class-to-prior lookup table to obtain physical priors.

For relatively uniform surfaces, such as indoor flooring or paved roads, assigning fixed priors to each semantic class is a reasonable approximation. Our work focuses on quadruped motion over complex off-road terrain. Even with correct segmentation, the lookup table assigns the same priors to all pixels of a class. For example, sparse and dense grass receive the same values if both are labeled as grass. PhyMap uses these labels as coarse supervision. At inference, it combines DINO, DPT, and geometric features to directly predict continuous, normalized physical priors. This allows the priors to vary within the same semantic region.

To illustrate this difference, we have added Figure 8 to Appendix C. It directly compares physical priors obtained from semantic masks with continuous PhyMap predictions. The examples include grass with wheel tracks and gravel mixed with vegetation. They show how the two methods represent complex ground surfaces.

Table 4 reports the results with the added baseline.

MethodBase ATE (m) ↓Vert. RMSE (m) ↓R/P RMSE (°) ↓Vel. RMSE (m/s) ↓
Geometry Only0.16830.08308.02151.2238
Uniform PhyMap0.14500.05004.82691.0850
ActiveSAM0.14270.03915.25821.1338
Full PhyMap0.13800.03783.35531.0246

Table 13 reports the comparison for long rollouts.

MethodATE @4.8s (m) ↓ATE @9.6s (m) ↓RotErr @9.6s (°) ↓Vel. RMSE @9.6s (m/s) ↓
A. Frozen Initial Map0.6171.34516.3070.534
B. Online Geo. + Frozen Physics0.6081.33413.7730.425
C. Online Geometry Only0.6531.40415.7810.392
D. Online Uniform PhyMap0.6041.26714.1980.359
ActiveSAM0.6121.34114.1260.387
E. Full DR-WM0.5681.24213.0960.322

In both evaluations, Full PhyMap and Full DR-WM achieve lower mean errors than the online ActiveSAM baseline on all four metrics. |

AuthorsOpenReview ↗

Response to Question 2

We sincerely thank you for your detailed feedback and helpful suggestions. We agree that online ActiveSAM is the most direct runtime baseline. We have therefore compared Phy-MapAnything with MapAnything + ActiveSAM. Both FPS values cover the complete inference pipeline, including geometry reconstruction and physical prior prediction. The ActiveSAM baseline runs online segmentation and uses the same class-to-prior lookup table.

MethodFull inference FPS ↑Mean peak GPU memory (GiB) ↓
Phy-MapAnything6.2316.23
MapAnything + ActiveSAM1.54215.19

Phy-MapAnything runs about four times as fast and uses less GPU memory. This directly compares our method with running ActiveSAM at inference time.

AuthorsOpenReview ↗

Response to Question 3

We greatly appreciate the care and attention you have given to our work. We have added Spatia as a baseline for 3D-controlled video generation in Section 4.2 of the revised manuscript. Tables 1 and 2 report the results. DR-WM performs better than Spatia on all reported metrics.

Table 1. VLM-based automatic evaluation of generated video quality on on-robot scenes.

MethodImaging ↑Aesthetic ↑Motion ↑Temp. Cons. ↑Action Adh. ↑Phys. Realism ↑
CogVideoX1.50.7360.7420.8260.7810.8390.793
Wan2.2-TI2V0.7790.7740.8590.8460.8840.842
WorldStereo0.7830.7630.8750.8610.8920.858
Matrix-Game 3.00.7890.7820.8860.8640.9130.870
Cosmos 30.7970.7910.8810.8730.9080.864
EgoWM0.6240.6530.7980.6690.8010.706
LingBot-World0.6730.6660.4190.7140.3340.328
NWM0.2430.2870.0940.1180.0410.069
Spatia0.7820.6910.8730.7970.9150.784
DR-WM (Ours)0.8140.8120.9240.8960.9490.913

Table 2. Reference-aligned evaluation of camera motion and video fidelity.

MethodRotErr (°) ↓ATE (m) ↓PSNR ↑LPIPS ↓DINO Sim. ↑Flow EPE ↓FVD-16 ↓
CogVideoX1.56.35720.49713.680.48100.608221.546738.37
Wan2.2-TI2V7.71711.86013.530.42720.699018.819683.42
WorldStereo7.0606.39513.240.48430.621322.199599.33
Matrix-Game 3.04.9231.70414.270.46230.69029.0531199.86
Cosmos 320.8053.65315.090.39870.744111.992801.40
EgoWM7.4691.07315.240.43170.63778.592824.32
LingBot-World13.16713.51413.580.50200.602116.3291148.22
NWM67.21714.85212.200.61510.207988.4691525.20
Spatia8.8851.21114.790.34800.75708.027620.63
DR-WM (Ours)3.9800.96117.010.28180.79936.001567.27
AuthorsOpenReview ↗

Response to Question 4

Thank you for taking the time to read our paper closely and share your valuable feedback. PhyMap predicts physical priors from visual features in a single forward pass. Its maps can have softer boundaries than maps based on semantic segmentation. These dense predictions can represent local terrain differences that matter for robot motion.

We added Figure 8 to Appendix C to compare continuous PhyMap predictions with priors obtained by applying a fixed lookup table to semantic masks. The examples include grass with wheel tracks and gravel mixed with vegetation. The lookup table assigns the same priors to all pixels in a semantic class. PhyMap can predict different values within that class based on local visual and geometric features. This ability is important for representing mixed surfaces in off-road terrain.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.