acceptodds
Under review as a conference paper at ICLR 2027

PhysWAM: Physically Consistent World–Action Modeling for Autonomous Driving

Abstract

World–action models (WAMs) jointly predict how a scene will evolve and how an agent should act, however joint generation alone does not necessarily impose a shared geometric constraint on these predictions. We present PhysWAM, a unified world–action model for autonomous driving that co-denoises multiview video, metric depth, and ego motion within a single flow-matching transformer. To ground world and action generation in measured scene geometry, we introduce Coupled Point Projection (CPP). CPP unprojects generated depth, transforms the resulting points using generated SE(3) ego motion, and penalizes their distance from LiDAR points transformed using recorded ego motion. This geometric loss promotes physical consistency with the measured scene by jointly supervising generated depth and motion alongside their standard flow-matching objectives. At inference, a simple label-free consensus rule selects among sampled trajectories without a learned scorer or simulator feedback. We evaluate PhysWAM across NAVSIM v1 and v2 planning, zero-shot closed-loop transfer, and future video and metric-depth prediction. Despite PhysWAM's simple selection procedure, it achieves strong planning performance and transfers robustly to unseen driving environments without additional training. It also generates accurate metric depth and temporally coherent video, with CPP improving both planning and depth prediction. Together, these results demonstrate that the geometric relationship between scene depth and ego motion provides a direct way to couple world and action generation within a simple unified model.

Primary area:
applications to robotics, autonomy, planning
Keywords:
autonomous driving, world models, world action models, video generation, end-to-end planning, flow matching, metric depth

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_5V4ZOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PhysWAM jointly generates multi-view future video, metric depth, and ego motion for autonomous driving. Its coupled point-projection loss (CPP) transforms predicted depth using predicted motion and compares the resulting 3D points with LiDAR-based references. Experiments on NAVSIM, Navhard, and HUGSIM show improvements in planning and generated geometry.

Critical strengths
  1. CPP provides a concrete geometric connection between generated depth and ego motion. Unlike feature conditioning alone, one metric residual directly supervises both outputs.
  2. The paper demonstrates downstream benefits across multiple driving benchmarks. Removing CPP reduces NAVSIM PDMS from 91.4 to 89.6 and HUGSIM HD from 35.5 to 33.4, supporting its practical effectiveness.
Critical weaknesses
  1. The distinctive contribution over existing geometry-aware world–action models, including GeoWorldAD, remains insufficiently isolated. The ablations show that adding CPP helps, but lack a matched comparison against explicit, separate metric-depth and composed-pose losses. Existing flow-matching and safety objectives are not equivalent controls.
  2. Each plan requires full future generation, costing 9.4 GPU-seconds at 30 sampling steps or 4.4 at eight steps. No action-only inference comparison establishes whether this expense is necessary for planning. Furthermore, CPP supervises agreement with recorded futures; consistency under counterfactual actions remains only indirectly evaluated.
Questions for authors
  1. How does CPP compare with separate metric-depth and composed-pose losses using the same supervision, training budget, and comparable gradient weighting?
  2. Can an action-only inference path preserve planning performance while avoiding future video and depth generation?
  3. What direct evidence supports depth–motion consistency under commands or trajectories that substantially differ from the recorded future?
7 comments

We thank the reviewer for a precise review. Each of the three questions is answered below by a new experiment, in the order asked. We have updated the paper with these results.

Summary

  • Q1 (matched comparison): we trained it. CPP outperforms the separate metric-depth and composed-pose losses on every metric at 4,000 updates: 80.3 against 78.2 PDMS, with fewer off-road and collision cases and a lower heading error (57.0 PDMS without geometric supervision).
  • Q2 (action-only inference): joint generation gives the best planner, and a single sampling pass keeps its planning score at 2.8 GPU-seconds instead of 9.4, without any few-step distillation. Run action-only, the same jointly trained model is 6 to 10% lower after a short fine-tune, at about 1.4 GPU-seconds.
  • Q3 (away from the recorded future): three direct tests. The generated depth follows the model's own motion in every deviation bin, including headings more than 10° off the recording and under swapped commands, and CPP makes the generated geometry more self-consistent.

Q1 / W1. Does CPP add anything over separate metric-depth and composed-pose losses?

Yes. Trained under identical conditions, CPP outperforms the separate losses on every metric.

We trained the comparison you describe: an explicit metric-depth loss (the metric error of the generated depth at each LiDAR-supported cell) and an explicit composed-pose loss (the error of the pose obtained by chaining the generated per-step motions, measured in metres at the LiDAR points), kept as two separate penalties. These are the depth-only and pose-only terms of App. A.2; CPP penalises their composition in one residual. The comparison holds fixed the three conditions raised in the review: same supervision (the same LiDAR-supported cells and the same recorded motion), same training budget (the same recipe, data, initialisation and number of updates), and comparable gradient weighting (each term is rescaled at every step to the gradient share that CPP receives).

Table 1. Training with CPP, without it, and with the separate losses (navtest, 4,000 updates, as in Table 3(a) of the paper; each row is the mean of two evaluation draws).

PDMS ↑EPDMS ↑Off-road ↓Collision ↓Heading error ↓
With CPP80.378.89.7%3.8%2.7°
Without CPP57.055.732.7%9.8%8.9°
Separate metric-depth and composed-pose losses78.277.011.4%4.8%4.1°

CPP is the strongest of the three on every column: against the separate losses it is 2.1 PDMS and 1.8 EPDMS higher, with a lower off-road rate, a lower collision rate and a lower heading error. Both forms of metric geometric supervision are far ahead of training without it (57.0 PDMS), so the supervision matters and composing depth with motion in one residual adds to it. Each arm takes about a day and a half on the GPUs available to us, so within the discussion period the control is evaluated at 4,000 updates, the budget at which Table 3(a) compares training with and without CPP.

Positioning against GeoWorldAD and related models. GeoWorldAD supervises geometry with separate terms: a camera-pose loss on the relative pose between observed frames, a depth loss and a point-map loss, plus a future-depth loss. Its point maps are expressed in the ego-camera frame of their own time step, so no term combines predicted depth with predicted motion; its planned trajectory receives a trajectory loss and a proposal-score loss and no geometric loss; and it does not generate video. In PhysWAM one metric residual is formed from the generated future depth moved by the generated future ego motion, which is the plan itself, and is compared with LiDAR, inside a model that generates video, depth and motion jointly and uses no learned scorer. CPP takes its inspiration from the classical idea of tying depth and camera motion through a single reprojection objective (SfMLearner, photometric and up to scale) and brings it into generative world-action models: a metric 3D residual against LiDAR, formed from the generated future depth and the generated plan. Table 1 shows that, on the same backbone, this composition outperforms the separate metric losses that prior geometry-aware models use.

Geometric supervisionActs on which motionOne residual of depth and motionGenerates video
GeoWorldADseparate terms: camera pose, depth, point maps; future depthrelative pose between observed framesnono
PhysWAM (CPP)one metric residual against LiDARthe generated future ego motion (the plan)yesyes

We have updated the related work to state this distinction precisely and to correct four statements: the description of what CPP adds over GeoWorldAD, GeoWAM and DVGT-2 (which already supervise ego pose in separate terms), the listing of X-WAM (a robot-manipulation model), the description of 4D-WAM's supervision, and the discussion of EponaV2.

Q2 / W2. Can an action-only inference path preserve planning while avoiding future video and depth generation?

Joint generation gives the best planner, and it needs far fewer steps than we used: a single sampling pass keeps the planning score at 2.8 GPU-seconds per plan. The same jointly trained model can also be run action-only, at about 1.4 GPU-seconds and 6 to 10% lower planning score.

We ran the comparison, and it separates two things: how many sampling steps a plan needs, and what generating the future contributes to it.

1. A plan does not need 30 sampling steps. The paper reports 4 to 30 steps. Going further, a single joint sampling step keeps 99.7% of the planning score of the default sampler, so a plan costs 2.8 GPU-seconds instead of 9.4. This is the trained model as it is, with no few-step distillation or fine-tuning.

Table 2. Sampler steps and the cost of a plan (navtest; PDMS normalized to the default 30-step sampler, the setting of all results in the paper; differences of less than 1 between rows are within sampling noise; cost of one plan on one RTX PRO 6000).

Joint generationSampler stepsNormalized PDMS (30 steps = 100)GPU-s / scene
Default sampler (Table 1 of the paper; one sample per scene)301009.4
8100.84.4
3100.43.2
2100.33.0
199.72.8

2. Generating the future improves the plan, and the same model can also be run action-only as a faster mode. We ran action-only inference in two ways: with the trained model as it is, and after fine-tuning it for action-only inference, where only the ego motion is denoised and no future video or depth is generated.

Table 3. Action-only inference against joint generation (navtest; same normalization; 2 / 8 / 30 sampler steps).

InferenceNormalized PDMS, 2 / 8 / 30 stepsGPU-s / scene, 2 / 8 / 30 steps
Joint generation of video, depth and motion100.3 / 100.8 / 1003.0 / 4.4 / 9.4
Action-only, trained model as it is67.3 / 76.2 / 74.61.4 / 1.5 / 2.5
Action-only, after 4,000 updates of fine-tuning93.8 / 90.0 / 90.41.4 / 1.5 / 2.5

Taking the future away from the trained model lowers the planning score to about three quarters, so the planner makes real use of the future it generates. With 4,000 updates of fine-tuning for action-only inference the model recovers to 90 to 94% at about 1.4 GPU-seconds per plan (still improving when we stopped), and joint generation remains the most accurate option.

Note. All rows of Table 3 use the full PhysWAM model, trained jointly on video, depth and motion with CPP; the action-only rows run that same model, as it is or after a short fine-tune. Training with future video generation is known to benefit the planner itself: on NAVSIM, Epona falls from 86.2 to 78.1 PDMS and DriveVLA-W0 from 83.9 to 70.3 when trained for the trajectory alone, without their future video prediction objective.

What this means for cost. With one-pass sampling or the action-only variant, a plan costs 3 to 7 times less than the 9.4 GPU-seconds of the baseline sampler. All costs above are for producing the plan. Decoding the generated video and depth with the VAE decoder is needed only to visualize the futures and adds 1.3 seconds per scene. Our implementation is a research prototype running on a single GPU; these results show how the cost can be brought down further, and reaching real-time rates is engineering work we leave to the future.

Reviewer_bdDbOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PhysWAM jointly predicts multiview future video, metric depth, and ego motion within a unified flow-matching model. The central contribution is Coupled Point Projection (CPP), which transforms points reconstructed from predicted depth using predicted ego motion and aligns them with LiDAR points transformed using recorded motion. This provides a shared geometric training signal for depth and action generation. The model is evaluated on NAVSIM planning, zero-shot closed-loop driving in HUGSIM, and future video and depth prediction. An optional medoid-based selection rule chooses a representative trajectory from multiple samples without a learned scorer.

Critical strengths
  1. CPP is a simple and well-motivated way to connect depth prediction with ego-motion generation. The full-model ablations show consistent improvements: CPP raises NAVSIM PDMS from 89.6 to 91.4, navhard EPDMS from 36.2 to 38.1, and HUGSIM HD-Score from 33.4 to 35.5. It also improves depth accuracy. These results support the usefulness of the proposed objective across several evaluation settings.

  2. The evaluation extends beyond planning scores to examine future depth, video quality, and agreement between generated modalities. The paper also reports inference cost and discusses its supervision requirements and limitations explicitly. Together with the full-model CPP ablation, these analyses provide a reasonably thorough assessment of the approach.

Critical weaknesses
  1. The main unresolved issue is whether the gains come specifically from coupling depth and motion, or from adding geometric supervision. Appendix A.2 derives depth-only and pose-only residuals, but does not compare their separate penalties against CPP experimentally. The existing flow-matching losses and safety hinges do not provide an equivalent control: latent velocity supervision differs from metric-depth supervision, while obstacle and drivable-area hinges differ from point-based supervision of composed poses. A matched decoupled baseline is therefore needed to substantiate the claimed advantage of coupling.

  2. The evidence for improved consistency between generated modalities is less complete than the evidence for improved planning. Table 15 reports consistency measurements only for the full model, without the corresponding CPP ablation. It is therefore difficult to determine how much CPP contributes to these measurements. Since its supervision is tied to the recorded future, its effect on consistency when the generated trajectory differs from the recorded one also remains unclear.

Questions for authors
  1. How does CPP compare with the sum of the separate depth-only and pose-only penalties described in Appendix A.2, using the same valid cells, noise weighting, and comparable gradient budgets? This comparison would help distinguish the benefit of coupling from that of additional geometric supervision.

  2. Could you provide the Table 15 consistency measurements for the model trained without CPP? Breaking these results down by the deviation between generated and recorded motion would also clarify whether the consistency gains extend beyond futures close to the recorded trajectory.

4 comments

We thank the reviewer for two precisely specified requests. We ran both as specified, and we have updated the paper with these results.

Summary

  • W1 / Q1 (matched decoupled baseline): we trained it as specified. CPP outperforms the sum of the separate depth-only and pose-only penalties on every metric at 4,000 updates: 80.3 against 78.2 PDMS, with fewer off-road and collision cases and a lower heading error (57.0 PDMS without geometric supervision).
  • W2 / Q2 (Table 15 without CPP, and by deviation): both are provided. CPP improves the agreement of the generated depths, during training and at the end, and its advantage holds for futures far from the recorded trajectory.

W1 / Q1. How does CPP compare with the sum of the separate depth-only and pose-only penalties of App. A.2?

Trained under identical conditions, CPP outperforms the separate penalties on every metric.

We agree that the existing flow-matching losses and hinges are not a control, and we trained the baseline as specified: the depth-only and pose-only penalties of App. A.2, ρδ(∥rD∥)+ρδ(∥rT∥)\rho_\delta(\lVert r_D \rVert) + \rho_\delta(\lVert r_T \rVert), in place of CPP's ρδ(∥rD+rT+rDT∥)\rho_\delta(\lVert r_D + r_T + r_{DT} \rVert), where ρδ\rho_\delta is the Huber penalty. The comparison holds fixed the conditions raised in the review: same valid cells (the same LiDAR-supported cells and anchors), same noise weighting (with the same Huber knee and reduction), and comparable gradient budgets (each term is rescaled at every step to the gradient share that CPP receives; the realised shares during training are 0.20 for both terms). The recipe, data, initialisation and number of updates are identical.

Table 1. Training with CPP, without it, and with the separate penalties (navtest, 4,000 updates, as in Table 3(a) of the paper; each row is the mean of two evaluation draws).

PDMS ↑EPDMS ↑Off-road ↓Collision ↓Heading error ↓
With CPP80.378.89.7%3.8%2.7°
Without CPP57.055.732.7%9.8%8.9°
Sum of the separate depth-only and pose-only penalties78.277.011.4%4.8%4.1°

CPP is the strongest of the three on every column: against the separate penalties it is 2.1 PDMS and 1.8 EPDMS higher, with a lower off-road rate, a lower collision rate and a lower heading error. Both forms of metric geometric supervision are far ahead of training without it (57.0 PDMS), so the supervision matters and composing depth with motion in one residual adds to it. Each arm takes about a day and a half on the GPUs available to us, so within the discussion period the control is evaluated at 4,000 updates, the budget at which Table 3(a) compares training with and without CPP.

W2 / Q2. How much does CPP contribute to the consistency measurements of Table 15, and do the gains extend beyond futures close to the recorded trajectory?

CPP improves the agreement of the generated depths, during training and at the end, and its advantage holds for futures far from the recorded trajectory.

Both requested results are below: Table 15 for the model trained without CPP, and the breakdown by the deviation between the generated and the recorded motion.

Table 2. The consistency table of the appendix (Table 15) for the model trained without CPP (end of training; the last column is the change when CPP is removed).

QuantityFloorPhysWAMChange without CPP
Yaw, video vs. motion, 4 s ↓0.29°0.80°-0.001°
Cross-view ∣Δlog⁡z∣\lvert \Delta \log z \rvert, +2 s, left / right ↓0.032 / 0.0300.045 / 0.049+0.0013 / +0.0004
Temporal ∣Δlog⁡z∣\lvert \Delta \log z \rvert, 0 to 4 s ↓0.0870.285-0.0005
Video follows the generated motion where it departs from the recorded one ↑67%-1.3 points

At the end of training, removing CPP raises the cross-view depth disagreement and lowers the share of scenes in which the video follows the generated motion; the yaw and temporal measures are unchanged.

By deviation between generated and recorded motion.

Table 3. Consistency by how far the generated motion departs from the recorded one (final models; each cell is with CPP / without CPP; medians). Heading deviation is the difference in heading between the generated and the recorded ego pose at +4 s; samples are generated futures, four per scene.

Heading deviationSamplesDepth fits its own motion ↑Cross-view depth disagreement ↓Video-motion yaw error ↓
under 2°3,1330.006 / 0.0050.047 / 0.0490.61° / 0.59°
2 to 10°1,4250.026 / 0.0270.045 / 0.0451.77° / 1.92°
over 10°2420.123 / 0.1080.043 / 0.0452.70° / 3.56°
  • Depth fits its own motion (↑): depth mismatch over 4 s when the generated depth is carried forward with the recorded motion, minus the mismatch when it is carried with the generated motion. Larger = the depth agrees more with the model's own motion (self-consistent).
  • Cross-view depth disagreement (↓): mismatch between the front-view depth projected into the side cameras and the depth generated for those views. Lower = the three views agree on one 3D scene.
  • Video-motion yaw error (↓): camera rotation estimated from the generated video, minus the rotation in the generated motion, over 4 s. Lower = the video shows the planned motion.

The further the model departs from the recording, the more clearly its depth goes with its own motion: the first column grows twenty-fold from the smallest to the largest deviation. Agreement across views is maintained at every deviation, and the yaw error rises for the larger departures, which involve larger turns.

During training. The contribution of CPP to the agreement of the generated depths is larger earlier in training.

Table 4. Self-consistency during training, with and without CPP (8,000 updates; measures as in Table 3, plus the depth disagreement between frames 4 s apart).

MeasureWith CPPWithout CPP
Cross-view depth disagreement ↓ (left / right view)0.053 / 0.0550.063 / 0.065
Depth disagreement over 4 s ↓0.2850.294
Video-motion yaw error ↓2.31°1.48°
Video-motion yaw error ↓, turns above 45°6.22°7.67°

CPP improves both depth measures, the cross-view one by 15% and in every deviation bin, and it already lowers the yaw error in sharp turns. Over all scenes the yaw error is still higher with CPP at this stage, likely because its geometric signal moves the motion toward the true geometry faster than the video follows early in training; by the end of training the two models reach the same value (0.80°). At the end of training the cross-view advantage remains, and where the generated trajectory departs most from the recording the model trained with CPP is ahead on all three measures of Table 3 (0.123 against 0.108, 0.043 against 0.045, and 2.70° against 3.56°).

CPP supervises consistency with the real world, and it also makes the generated geometry more self-consistent, including far from the recorded trajectory. Supervising self-consistency directly is the natural next step (see limitations).

We hope these results resolve the reviewer's concerns, and we would be grateful if the reviewer would take them into account in the final assessment. We are glad to run anything further during the discussion.

Hi reviewer bdDb, we wanted to reach out if our response addresses your concerns.

Happy to provide any further clarifications if needed, please let us know.

Thanks

Area_Chair_oNCXOpenReview ↗

Please help to join the discussion

Dear Reviewer bdDb,

The discussion deadline is approaching. Please help to join the discussion and offer the reviewer's opportunity to further respond to your concerns.

Thanks!

AC

Reviewer_7jAtOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PhysWAM, a unified world-action model that jointly generates future multi-view video, metric depth, and ego motion using a flow-matching transformer. Its main contribution is Coupled Point Projection (CPP), which transforms predicted depth into 3D points using generated ego motion and compares them with LiDAR geometry transformed using recorded motion. This provides joint geometric supervision for depth and motion prediction. The paper also uses obstacle and drivable-area constraints and a simple medoid-based trajectory selection strategy. Experiments on NAVSIM and HUGSIM show competitive planning performance, improved depth prediction, and promising zero-shot closed-loop transfer.

Critical strengths
  • The idea of jointly constraining generated depth and ego motion through a shared geometric objective is reasonable and well-motivated.
  • PhysWAM achieves competitive results on NAVSIM and transfers to HUGSIM without target-domain training. It also reports future video, depth, and geometric consistency metrics rather than focusing only on planning.
Critical weaknesses
  • PhysWAM generates future RGB video, depth, and ego motion together, even though planning only requires the ego trajectory. The reported inference cost seems expensive for real-time driving. Moreover, using eight samples for medoid selection brings only small improvements (0.3 PDMS and 0.1 EPDMS). Could the authors evaluate whether ego motion can be generated without generating the full video and depth?

  • The paper provides clear evidence that CPP improves planning and depth prediction compared with standard flow-matching supervision. However, it is still unclear whether these improvements mainly come from this coupling mechanism or from introducing additional supervision in metric 3D space. For example, directly supervising the predicted depth against LiDAR and the predicted ego pose against recorded motion could also provide useful geometric gradients without coupling the two predictions.

Questions for authors
  • CPP uses the recorded future depth and ego motion for supervision. However, the model can generate trajectories that differ from the recorded one. It would be useful to discuss whether the generated depth and motion remain consistent in these cases, particularly when the generated trajectory differs substantially from the recorded trajectory.

  • Table 3(c) compares PhysWAM's depth errors on NAVSIM with results from other methods evaluated on nuScenes. Although the difference in evaluation settings is acknowledged, these numbers are not directly comparable. A clearer separation between within-dataset comparisons and cross-dataset reference results would avoid potentially misleading conclusions.

  • The introduction is too long and the content before the proposed method can be more concise to improve the overall clarity and readability of the paper.

7 comments

We thank the reviewer for the careful reading. We ran the experiments asked for, we agree with the presentation points, and we have updated the paper with these results.

Summary

  • W1 (cost; ego motion without the full video and depth): joint generation gives the best planner, and a single sampling pass keeps its planning score at 2.8 GPU-seconds instead of 9.4, without any few-step distillation. Run action-only, the same jointly trained model is 6 to 10% lower after a short fine-tune, at about 1.4 GPU-seconds. Medoid selection is an optional add-on; PhysWAM uses no learned trajectory scorer.
  • W2 (coupling or supervision): we trained the control. CPP outperforms separate depth and pose supervision on every metric at 4,000 updates: 80.3 against 78.2 PDMS, with fewer off-road and collision cases and a lower heading error (57.0 PDMS without geometric supervision).
  • Q1 (when the generated trajectory differs from the recorded one): the generated depth follows the model's own motion in every deviation bin, including headings more than 10° off the recording, and CPP makes the generated geometry more self-consistent.
  • Q2, Q3: agreed and done. Table 3(c) is separated and has a same-dataset reference, and we have refined the introduction.

W1. Can the ego motion be generated without the full video and depth, and what does a plan cost?

Joint generation gives the best planner, and it needs far fewer steps than we used: a single sampling pass keeps the planning score at 2.8 GPU-seconds per plan. The same jointly trained model can also be run action-only, at about 1.4 GPU-seconds and 6 to 10% lower planning score.

We ran the comparison, and it separates two things: how many sampling steps a plan needs, and what generating the future contributes to it.

1. A plan does not need 30 sampling steps. The paper reports 4 to 30 steps. Going further, a single joint sampling step keeps 99.7% of the planning score of the default sampler, so a plan costs 2.8 GPU-seconds instead of 9.4. This is the trained model as it is, with no few-step distillation or fine-tuning.

Table 1. Sampler steps and the cost of a plan (navtest; PDMS normalized to the default 30-step sampler, the setting of all results in the paper; differences of less than 1 between rows are within sampling noise; cost of one plan on one RTX PRO 6000).

Joint generationSampler stepsNormalized PDMS (30 steps = 100)GPU-s / scene
Default sampler (Table 1 of the paper; one sample per scene)301009.4
8100.84.4
3100.43.2
2100.33.0
199.72.8

2. Generating the future improves the plan, and the same model can also be run action-only as a faster mode. We ran action-only inference in two ways: with the trained model as it is, and after fine-tuning it for action-only inference, where only the ego motion is denoised and no future video or depth is generated.

Table 2. Action-only inference against joint generation (navtest; same normalization; 2 / 8 / 30 sampler steps).

InferenceNormalized PDMS, 2 / 8 / 30 stepsGPU-s / scene, 2 / 8 / 30 steps
Joint generation of video, depth and motion100.3 / 100.8 / 1003.0 / 4.4 / 9.4
Action-only, trained model as it is67.3 / 76.2 / 74.61.4 / 1.5 / 2.5
Action-only, after 4,000 updates of fine-tuning93.8 / 90.0 / 90.41.4 / 1.5 / 2.5

Taking the future away from the trained model lowers the planning score to about three quarters, so the planner makes real use of the future it generates. With 4,000 updates of fine-tuning for action-only inference the model recovers to 90 to 94% at about 1.4 GPU-seconds per plan (still improving when we stopped), and joint generation remains the most accurate option.

Note. All rows of Table 2 use the full PhysWAM model, trained jointly on video, depth and motion with CPP; the action-only rows run that same model, as it is or after a short fine-tune. Training with future video generation is known to benefit the planner itself: on NAVSIM, Epona falls from 86.2 to 78.1 PDMS and DriveVLA-W0 from 83.9 to 70.3 when trained for the trajectory alone, without their future video prediction objective.

What this means for cost. With one-pass sampling or the action-only variant, a plan costs 3 to 7 times less than the 9.4 GPU-seconds of the baseline sampler. All costs above are for producing the plan. Decoding the generated video and depth with the VAE decoder is needed only to visualize the futures and adds 1.3 seconds per scene. Our implementation is a research prototype running on a single GPU; these results show how the cost can be brought down further, and reaching real-time rates is engineering work we leave to the future.

Medoid selection. The medoid rule is an optional, training-free add-on, and none of the main results depend on it. PhysWAM reaches its planning score with one sample per scene and without any learned trajectory scorer, whereas several strong methods on this benchmark score many candidate trajectories with a scorer trained on simulator metrics (GeoWorldAD, for example, scores 64 proposals). The headroom for selection is large: the per-scene best of eight samples is far above the single-sample score (oracle-of-8 row in Table 1 of the paper). The medoid captures a small part of it, 0.1 to 0.7 PDMS across 3 to 8 samples.

W2. Do the improvements come from the coupling, or from additional supervision in metric 3D space?

Both contribute, and CPP, which couples the two, is the best on every metric.

We trained the control you describe: the generated depth supervised against LiDAR and the generated ego pose supervised against the recorded motion, as two separate penalties. Everything else is identical to CPP: the same cells, the same noise weighting, the same gradient weighting, and the same recipe, data, initialisation and number of updates.

Table 3. Training with CPP, without it, and with the separate penalties (navtest, 4,000 updates, as in Table 3(a) of the paper; each row is the mean of two evaluation draws).

PDMS ↑EPDMS ↑Off-road ↓Collision ↓Heading error ↓
With CPP80.378.89.7%3.8%2.7°
Without CPP57.055.732.7%9.8%8.9°
Separate depth and pose supervision78.277.011.4%4.8%4.1°

CPP is the strongest of the three on every column: against the separate penalties it is 2.1 PDMS and 1.8 EPDMS higher, with a lower off-road rate, a lower collision rate and a lower heading error. Both forms of metric geometric supervision are far ahead of training without it (57.0 PDMS), so the supervision matters and composing depth with motion in one residual adds to it. Each arm takes about a day and a half on the GPUs available to us, so within the discussion period the control is evaluated at 4,000 updates, the budget at which Table 3(a) compares training with and without CPP.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Summary of Reviewer Concerns + Updated Rebuttal Paper Draft

We thank the three reviewers for their careful and constructive reviews. The reviews converge on three questions, and we ran every experiment that was asked for. The results are summarised here and given in full in the individual responses; we have updated the paper with them. All changes are marked in blue in the revised PDF.

1. Is the gain from coupling depth with motion, or from separate metric supervision? (7jAt-W2, bdDb-W1/Q1, 5V4Z-W1/Q1)

Trained under identical conditions, CPP outperforms the separate depth and pose penalties on every metric. The control uses the depth-only and pose-only penalties of App. A.2 on the same cells, with the same noise weighting and the same gradient budgets.

navtest, 4,000 updatesPDMS ↑EPDMS ↑Off-road ↓Collision ↓Heading error ↓
With CPP80.378.89.7%3.8%2.7°
Without CPP57.055.732.7%9.8%8.9°
Separate depth and pose penalties78.277.011.4%4.8%4.1°

2. Is generating the future necessary for the plan, and what does a plan cost? (7jAt-W1, 5V4Z-W2/Q2)

Joint generation gives the best planner, and it needs far fewer sampling steps than we used. A single joint sampling step keeps 99.7% of the planning score at 2.8 GPU-seconds per plan instead of 9.4, without any few-step distillation or fine-tuning. The same jointly trained model can also be run action-only, at about 1.4 GPU-seconds and 6 to 10% lower planning score.

3. Are the generated depth, video and motion consistent away from the recorded future? (7jAt-Q1, bdDb-W2/Q2, 5V4Z-W2/Q3)

Yes. The generated depth follows the model's own motion in every deviation bin, including headings more than 10° off the recording and under swapped driving commands, and CPP makes the generated geometry more self-consistent (15% lower cross-view depth disagreement at 8,000 updates). We also provide the appendix consistency table for the model trained without CPP, as requested.

4. Presentation (7jAt-Q2, Q3; 5V4Z-W1)

Table 3(c) now separates within-dataset rows from values reported on nuScenes and has a same-dataset reference; we have refined the introduction; and the related work positions CPP against GeoWorldAD and other geometry-aware models.

We believe these experiments have helped us improve the quality of the paper: CPP is tested against the control the reviewers asked for and demonstrates strong performance across metrics, a plan can cost a third as much as with the baseline sampler at similar performance, and the consistency of the generated futures is now measured directly away from the recorded trajectory. We appreciate the reviewers' time and effort in providing valuable feedback.

Area_Chair_oNCXOpenReview ↗

Reviewers, please help to join the discussion.

Dear reviewers,

The authors have uploaded the rebuttal. Please help to join the discussion and the further process.

Thanks!

Your AC

Hi reviewers, please let us know if there are any additional clarifications/concerns we can address, appreciate the time and effort spend during the review which helped us improve the paper quality,

Thanks

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.