- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
PhysWAM jointly generates multi-view future video, metric depth, and ego motion for autonomous driving. Its coupled point-projection loss (CPP) transforms predicted depth using predicted motion and compares the resulting 3D points with LiDAR-based references. Experiments on NAVSIM, Navhard, and HUGSIM show improvements in planning and generated geometry.
Critical strengths
- CPP provides a concrete geometric connection between generated depth and ego motion. Unlike feature conditioning alone, one metric residual directly supervises both outputs.
- The paper demonstrates downstream benefits across multiple driving benchmarks. Removing CPP reduces NAVSIM PDMS from 91.4 to 89.6 and HUGSIM HD from 35.5 to 33.4, supporting its practical effectiveness.
Critical weaknesses
- The distinctive contribution over existing geometry-aware world–action models, including GeoWorldAD, remains insufficiently isolated. The ablations show that adding CPP helps, but lack a matched comparison against explicit, separate metric-depth and composed-pose losses. Existing flow-matching and safety objectives are not equivalent controls.
- Each plan requires full future generation, costing 9.4 GPU-seconds at 30 sampling steps or 4.4 at eight steps. No action-only inference comparison establishes whether this expense is necessary for planning. Furthermore, CPP supervises agreement with recorded futures; consistency under counterfactual actions remains only indirectly evaluated.
Questions for authors
- How does CPP compare with separate metric-depth and composed-pose losses using the same supervision, training budget, and comparable gradient weighting?
- Can an action-only inference path preserve planning performance while avoiding future video and depth generation?
- What direct evidence supports depth–motion consistency under commands or trajectories that substantially differ from the recorded future?
7 comments
Q1 / W1. Does CPP add anything over separate metric-depth and composed-pose losses?
Yes. Trained under identical conditions, CPP outperforms the separate losses on every metric.
We trained the comparison you describe: an explicit metric-depth loss (the metric error of the generated depth at each LiDAR-supported cell) and an explicit composed-pose loss (the error of the pose obtained by chaining the generated per-step motions, measured in metres at the LiDAR points), kept as two separate penalties. These are the depth-only and pose-only terms of App. A.2; CPP penalises their composition in one residual. The comparison holds fixed the three conditions raised in the review: same supervision (the same LiDAR-supported cells and the same recorded motion), same training budget (the same recipe, data, initialisation and number of updates), and comparable gradient weighting (each term is rescaled at every step to the gradient share that CPP receives).
Table 1. Training with CPP, without it, and with the separate losses (navtest, 4,000 updates, as in Table 3(a) of the paper; each row is the mean of two evaluation draws).
| PDMS ↑ | EPDMS ↑ | Off-road ↓ | Collision ↓ | Heading error ↓ | |
|---|---|---|---|---|---|
| With CPP | 80.3 | 78.8 | 9.7% | 3.8% | 2.7° |
| Without CPP | 57.0 | 55.7 | 32.7% | 9.8% | 8.9° |
| Separate metric-depth and composed-pose losses | 78.2 | 77.0 | 11.4% | 4.8% | 4.1° |
CPP is the strongest of the three on every column: against the separate losses it is 2.1 PDMS and 1.8 EPDMS higher, with a lower off-road rate, a lower collision rate and a lower heading error. Both forms of metric geometric supervision are far ahead of training without it (57.0 PDMS), so the supervision matters and composing depth with motion in one residual adds to it. Each arm takes about a day and a half on the GPUs available to us, so within the discussion period the control is evaluated at 4,000 updates, the budget at which Table 3(a) compares training with and without CPP.
Positioning against GeoWorldAD and related models. GeoWorldAD supervises geometry with separate terms: a camera-pose loss on the relative pose between observed frames, a depth loss and a point-map loss, plus a future-depth loss. Its point maps are expressed in the ego-camera frame of their own time step, so no term combines predicted depth with predicted motion; its planned trajectory receives a trajectory loss and a proposal-score loss and no geometric loss; and it does not generate video. In PhysWAM one metric residual is formed from the generated future depth moved by the generated future ego motion, which is the plan itself, and is compared with LiDAR, inside a model that generates video, depth and motion jointly and uses no learned scorer. CPP takes its inspiration from the classical idea of tying depth and camera motion through a single reprojection objective (SfMLearner, photometric and up to scale) and brings it into generative world-action models: a metric 3D residual against LiDAR, formed from the generated future depth and the generated plan. Table 1 shows that, on the same backbone, this composition outperforms the separate metric losses that prior geometry-aware models use.
| Geometric supervision | Acts on which motion | One residual of depth and motion | Generates video | |
|---|---|---|---|---|
| GeoWorldAD | separate terms: camera pose, depth, point maps; future depth | relative pose between observed frames | no | no |
| PhysWAM (CPP) | one metric residual against LiDAR | the generated future ego motion (the plan) | yes | yes |
We have updated the related work to state this distinction precisely and to correct four statements: the description of what CPP adds over GeoWorldAD, GeoWAM and DVGT-2 (which already supervise ego pose in separate terms), the listing of X-WAM (a robot-manipulation model), the description of 4D-WAM's supervision, and the discussion of EponaV2.
Q2 / W2. Can an action-only inference path preserve planning while avoiding future video and depth generation?
Joint generation gives the best planner, and it needs far fewer steps than we used: a single sampling pass keeps the planning score at 2.8 GPU-seconds per plan. The same jointly trained model can also be run action-only, at about 1.4 GPU-seconds and 6 to 10% lower planning score.
We ran the comparison, and it separates two things: how many sampling steps a plan needs, and what generating the future contributes to it.
1. A plan does not need 30 sampling steps. The paper reports 4 to 30 steps. Going further, a single joint sampling step keeps 99.7% of the planning score of the default sampler, so a plan costs 2.8 GPU-seconds instead of 9.4. This is the trained model as it is, with no few-step distillation or fine-tuning.
Table 2. Sampler steps and the cost of a plan (navtest; PDMS normalized to the default 30-step sampler, the setting of all results in the paper; differences of less than 1 between rows are within sampling noise; cost of one plan on one RTX PRO 6000).
| Joint generation | Sampler steps | Normalized PDMS (30 steps = 100) | GPU-s / scene |
|---|---|---|---|
| Default sampler (Table 1 of the paper; one sample per scene) | 30 | 100 | 9.4 |
| 8 | 100.8 | 4.4 | |
| 3 | 100.4 | 3.2 | |
| 2 | 100.3 | 3.0 | |
| 1 | 99.7 | 2.8 |
2. Generating the future improves the plan, and the same model can also be run action-only as a faster mode. We ran action-only inference in two ways: with the trained model as it is, and after fine-tuning it for action-only inference, where only the ego motion is denoised and no future video or depth is generated.
Table 3. Action-only inference against joint generation (navtest; same normalization; 2 / 8 / 30 sampler steps).
| Inference | Normalized PDMS, 2 / 8 / 30 steps | GPU-s / scene, 2 / 8 / 30 steps |
|---|---|---|
| Joint generation of video, depth and motion | 100.3 / 100.8 / 100 | 3.0 / 4.4 / 9.4 |
| Action-only, trained model as it is | 67.3 / 76.2 / 74.6 | 1.4 / 1.5 / 2.5 |
| Action-only, after 4,000 updates of fine-tuning | 93.8 / 90.0 / 90.4 | 1.4 / 1.5 / 2.5 |
Taking the future away from the trained model lowers the planning score to about three quarters, so the planner makes real use of the future it generates. With 4,000 updates of fine-tuning for action-only inference the model recovers to 90 to 94% at about 1.4 GPU-seconds per plan (still improving when we stopped), and joint generation remains the most accurate option.
Note. All rows of Table 3 use the full PhysWAM model, trained jointly on video, depth and motion with CPP; the action-only rows run that same model, as it is or after a short fine-tune. Training with future video generation is known to benefit the planner itself: on NAVSIM, Epona falls from 86.2 to 78.1 PDMS and DriveVLA-W0 from 83.9 to 70.3 when trained for the trajectory alone, without their future video prediction objective.
What this means for cost. With one-pass sampling or the action-only variant, a plan costs 3 to 7 times less than the 9.4 GPU-seconds of the baseline sampler. All costs above are for producing the plan. Decoding the generated video and depth with the VAE decoder is needed only to visualize the futures and adds 1.3 seconds per scene. Our implementation is a research prototype running on a single GPU; these results show how the cost can be brought down further, and reaching real-time rates is engineering work we leave to the future.
We thank the reviewer for a precise review. Each of the three questions is answered below by a new experiment, in the order asked. We have updated the paper with these results.
Summary