- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes AD-E2E-JEPA, an autonomous driving planning method based on an action-conditioned latent world model. Unlike conventional end-to-end approaches that directly train trajectory planners through imitation learning, this method first trains a world model through self-supervised learning to capture the relationship between ego actions and future visual representations. Specifically, the method employs DINOv3 to extract visual features, introduces a convolutional projector to compress feature dimensions, and applies SIGReg regularization to prevent representation collapse. Experiments on NAVSIMv2 demonstrate approximately 100× faster planning compared to existing latent world model-based methods.
Critical strengths
-
This paper explores an autonomous driving planning approach that directly selects trajectories using an action-conditioned world model, without requiring a dedicated planner trained through imitation learning. This is a meaningful research direction that provides an alternative perspective on the application of world models to autonomous driving.
-
By compressing visual features, the proposed method substantially reduces the computational cost of world model-based trajectory evaluation, achieving approximately 100× speedup while maintaining competitive planning performance.
Critical weaknesses
-
In Table 3, although the pretrained projector improves EPDMS from 80.2 to 85.4, the resulting performance remains below several existing methods. Moreover, the compared methods use different input configurations and supervision settings (SV/MV, PF/PB), which directly causes that we cannot make any fair comparison to existing methods. While the authors present these methods as references, the only controlled baseline is a randomly initialized projector, which is insufficient to establish the advantages of the proposed world model pretraining over alternative pretraining strategies. Therefore, the downstream experiments provide limited evidence of the competitiveness of the learned representations.
-
The paper lacks critical ablation studies on its core components, particularly the convolutional projector and SIGReg regularization. The authors do not investigate different feature compression ratios, simpler projector architectures, or the effect of removing SIGReg. Therefore, it is difficult to determine the individual contributions of these components to the reported performance and efficiency improvements. Furthermore, the experimental section is confusing and does not clearly distinguish the main comparisons from ablation studies. In particular, Table 2 mixes comparisons with existing world models and analyses of rollout training, training data scale, and the number of candidate trajectories, making the purpose and key findings of each experiment unclear.
Questions for authors
-
In Table 3, the proposed method achieves 85.4 EPDMS, which remains below several existing methods. Could the authors further explain the performance gap and clarify the motivation for including methods with different input configurations and supervision settings (SV/MV, PF/PB)?
-
Table 3 only compares the pretrained projector against random initialization. Could the authors provide comparisons with alternative pretraining strategies to better demonstrate the specific benefits of action-conditioned world model pretraining?
-
Could the authors provide additional ablation studies on the core components, particularly the effect of removing SIGReg, varying feature compression ratios, and replacing the convolutional projector with simpler alternatives such as average pooling?
We thank the Reviewer for the feedback. We have incorporated editions into our current draft and respond below.
Q1 & Q2 (and Weakness 1):
The primary contribution of our paper is JEPA-based world-model planning, while our downstream imitation-learning experiments demonstrate that the pretrained representations also benefit end-to-end driving without requiring future images at inference time. As mentioned in Section 4.4, existing imitation-learning methods are included in Table 3 for reference only, as they differ in experimental settings and some require industry-level computational resources, such as 32–64 A800 GPUs for several days. In contrast, our downstream experiments use only 2 A100 GPUs for 13 hours. The most direct controlled comparison is therefore between our pretrained projector and a randomly initialized projector under identical settings, as reported in Table 3, where EPDMS improves from 80.2 to 85.4.
Q3 (and Weakness 2)
We conducted additional ablation studies to evaluate different feature compression methods while reporting the shapes of the per-frame embeddings used for planning: (1) Learnable global projector, which adds additional layers on top of our learnable patch projector to further compress the patch embeddings into a single 256-dimensional vector per frame; (2) 2D average pooling (D=1024), which directly reduces the spatial dimensions of the DINO features while preserving the original embedding dimension of 1024; (3) 2D average pooling (D=256), which first reduces the spatial dimensions of the DINO features while preserving the original embedding dimension of 1024, then averages groups of four features to reduce the embedding dimension to 256; and (4) Learnable patch projector without SIGReg, which uses our learnable patch projector without SIGReg regularization.
Ablation study of feature compression methods on 100 NAVSIMv2 navtest scenes. FDE is measured at the 4-second planning horizon. These results demonstrate that our learnable patch projector achieves the best driving performance and the lowest reported final displacement error, supporting the effectiveness of the proposed compression design.
We thank the reviewer for requesting channel and spatial compression ablations.
The existing DINO-WM/JEPA-WM models, with a planning embedding shape of (32, 16, 1024), can also be viewed as an ablation of spatial compression and dimensionality reduction relative to our proposed projector, which produces a compressed embedding shape of (4, 8, 256).
Due to time constraints, we trained each AD-E2E-JEPA projector variant for 5 epochs (navtrain, no rollout) to additionally compare channel and spatial ablations. Results:
These preliminary results suggest that spatial compression and channel can further improve planning performance. Full training was incomplete at the rebuttal deadline; we plan to report the results in the camera-ready version if the paper is accepted.