- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how sparse future visual prediction can support autonomous-driving trajectory planning. DriveWeave jointly denoises trajectory, auxiliary future ego-state, and multi-view visual latents using a pretrained video DiT. A trajectory codec aligns latent distributions, shared attention exchanges information, and a pose-consistency energy is applied during both training and sampling. The default model predicts one terminal visual anchor while retaining all eight waypoints over four seconds. On the original NAVSIM navtest protocol, it reports 91.3 PDMS. Ablations separate information exchange, consistency training, sampling correction, and visual-target density.
Critical strengths
-
The paper provides a concrete way to coordinate future visual prediction and trajectory planning. Shared attention exchanges information, while a common ego-pose representation allows the model to compare what the predicted trajectory and imagined observations imply about motion. Applying this consistency objective during both training and sampling is a sensible design. Table 4 supports the distinction: consistency training improves PDMS from 87.9 to 90.0 without sampling correction, and the full method reaches 91.3. This shows that the benefit extends beyond an additional inference-time adjustment.
-
Sparse future imagination offers a useful accuracy–efficiency trade-off. The model retains all eight waypoints over a four-second horizon while predicting only a terminal multi-view visual anchor. In Table 3, this achieves 91.3 PDMS, compared with 85.1 for plan-only prediction and 90.1 for dense visual prediction; the reported latency falls from 340.2 ms for dense prediction to 66.2 ms. These results provide useful evidence that planning can benefit from a compact future visual target without paying the cost of a dense visual rollout.
Critical weaknesses
-
The evidence for visual–trajectory consistency relies heavily on the same learned pose readout used to enforce that consistency. The frozen readout R_eta converts generated visual latents into an ego-pose estimate, and both the consistency loss and the reported E_pose use this estimate (Secs. 3.3 and 4.1). A lower E_pose therefore establishes agreement under the training proxy, but does not independently verify that the generated views depict the motion specified by the trajectory. The PDMS improvements remain meaningful; what is less well established is the visual-consistency explanation for them. The paper refers to a future-image shuffling test in Appendix D, but its results and the readout's held-out pose accuracy are not available in the supplied PDF. Validation of this central component is needed to support the claimed geometric coordination.
-
The contribution of trajectory–visual distribution alignment is not isolated experimentally. ActionConvAE combines trajectory reconstruction with an MMD objective, and the paper argues that bringing trajectory latents closer to visual latents helps reuse the pretrained video backbone (Sec. 3.2). However, Fig. 4(a) shows marginal density profiles, while the planning ablations in Tables 2–4 keep the codec fixed. These results do not establish whether MMD-based alignment improves planning over a reconstruction-only codec, or whether the gain comes simply from using a learned trajectory representation. A controlled codec comparison, reporting both waypoint reconstruction accuracy and downstream PDMS, is needed to substantiate this part of the method.
Questions for authors
-
How accurately does the frozen visual-motion readout recover ego pose on held-out recorded image pairs, and what does the future-image shuffling test show? For generated views, does the reduction in E_pose also correspond to better visual–trajectory agreement under an independent motion estimate? Please provide the relevant Appendix D results and distinguish readout accuracy from agreement among the model's own predictions.
-
How does ActionConvAE with MMD compare with the same codec trained only for reconstruction, keeping the DiT training and inference settings fixed? Please report waypoint position/heading reconstruction errors and downstream PDMS. This would clarify whether matching the visual latent distribution provides a planning benefit beyond the learned trajectory encoding itself.
2 comments
Thank you for the detailed response. The held-out pose results and the shuffling test address my questions about what the readout learns, and I appreciate the independent evaluation on generated views. It would help to report the error against the ground-truth endpoint separately from the gap between the visually estimated and planned endpoints, since the latter is the agreement I was asking about.
The comparison with and without MMD addresses my main concern about the codec. The PDMS drop from 91.3 to 90.2 with the same ActionConvAE provides evidence that MMD contributes beyond the learned trajectory representation. Reporting position and heading reconstruction errors for both variants would make the comparison more complete. Overall, the response addresses most of my concerns, and I am keeping my rating.
Response to 7PWF: Pose Consistency, MMD Alignment, and Generation Validity
Thank you for your recognition of our paper,your valuable comments greatly improved its quality.
W1/Q1: Table S3 obtains 0.47 m/2.8° on held-out real image pairs; shuffling future images raises the error to 2.36 m/11.7°, indicating the readout uses future-image motion information and pose accuracy is guaranteed. Your concern may be robustness. However, Ev itself is generated and contains errors far larger than the readout error, so a slightly worse readout has little effect. The first consistency is pose-consistency(E_pose) term, acting on the sampling-denoising formula, optimizing both the denoising velocity field and the added consistency cost, so the model understands multimodal denoising. We also use this as a simple evaluation metric to assess the validity of generation. Existing generative backbones still fit single-modality generation; for multimodality they treat visual and action tokens as the same kind tokens for denoising. We want more than per-token imitating GT: even with low MSE loss, one cannot tell which tokens are good, and action results may still be poor. We internalize autonomous-driving requirements—unified heading and observation angles—into the reverse dynamics of denoising, adding constraints between tokens for more effective multimodal generation. Ev-GT and Ea-GT correctness is achieved by optimizing MSE loss during training so they learn correct results. Pose-consistency guarantees valid generation, not correct generation. Table S4 uses a visual pose estimator independent of training/sampling guidance to evaluate generated images' 4-s endpoint position; independent evaluation also shows improvement relative to GT.
W2/Q2-Revision: Added a simpler tiling results row to Table S5. Table S5 already isolates the MMD increment: keeping ActionConvAE and the reconstruction objective but removing MMD drops PDMS from 91.3 to 90.2 and raises the independently estimated visual–trajectory endpoint difference from 2.35 m to 2.55 m, supporting “adding distribution matching on this codec is beneficial.” The fixed-tiling row shows a sharp drop, clearly demonstrating this distribution matching’s role. The second 'consistency' is alignment(MMD) term, acting on information-interaction understanding. This is a problem we found during training: in the early stage, trajectory generation was already good, but image quality still did not meet requirements. This indicates trajectory denoising did not well use visual-generation priors; the two were largely generated separately. Alignment makes information interaction between denoising directions and velocities of trajectories and images more effective, enabling better exploit information from both. Lifting the eight-segment polyline to the same feature distribution as images lets the model learn what it should look like as observations change from current to future. Although supervising directly 4s ahead is difficult, it is necessary: a long trajectory helps the world model better understand evolution rules, whereas small segment-by-segment supervision may merely imitate. Pose-consistency(E_pose) ensures valid generation in sampling; alignment(MMD) ensures sufficient information interaction to learn correct generation in training.