acceptodds
Under review as a conference paper at ICLR 2027

AD-E2E-JEPA: A Joint-Embedding Predictive Architecture For End-to-End Autonomous Driving

10 views

Abstract

Autonomous driving requires that can understand the physical world, reason and plan, and operate safely. In this paper, we first systematically evaluate existing action-conditioned joint-embedding predictive architecture (JEPA) world models, including LeWM, DINO-WM, and JEPA-WM for end-to-end autonomous driving (E2EAD). To isolate world-model quality from policy learning, we employ a goal-conditioned zero-shot planning setting that evaluates these models using ground-truth future observations as goals, without training any driving policy. We find that existing JEPA-based world models are either accurate for driving but computationally expensive, or computationally efficient but insufficient for planning. To address this trade-off, we propose , which introduces a SIGReg-regularized learnable projector applied to projected patch embeddings. The projector reduces the number of planning patches by and the embedding dimension by , achieving a inference speedup while retaining planning performance, with a 0.8-second runtime for an 8-frame rollout over 256 candidate trajectories. Without training any driving policy, the world model itself reaches the goals located 20 meters away on average within the displacement of respectively 4.0/3.0 meters, using world-model rollouts over trajectory vocabularies of respectively 256/2,048 candidates. On the NAVSIMv2 benchmark, it achieves 67.3/71.5 EPDMS with multiplicative safety metrics and 84.1/86.0 EPDMS without them in goal-conditioned zero-shot planning. Experiments further show that the self-supervised pretrained projector improves downstream imitation learning performance from 80.2 to 85.4 EPDMS.

Primary area:
applications to robotics, autonomy, planning
Keywords:
end-to-end autonomous driving, world models, joint-embedding predictive architecture, self-supervised learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_SLMTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes AD-E2E-JEPA, an autonomous driving planning method based on an action-conditioned latent world model. Unlike conventional end-to-end approaches that directly train trajectory planners through imitation learning, this method first trains a world model through self-supervised learning to capture the relationship between ego actions and future visual representations. Specifically, the method employs DINOv3 to extract visual features, introduces a convolutional projector to compress feature dimensions, and applies SIGReg regularization to prevent representation collapse. Experiments on NAVSIMv2 demonstrate approximately 100× faster planning compared to existing latent world model-based methods.

Critical strengths
  1. This paper explores an autonomous driving planning approach that directly selects trajectories using an action-conditioned world model, without requiring a dedicated planner trained through imitation learning. This is a meaningful research direction that provides an alternative perspective on the application of world models to autonomous driving.

  2. By compressing visual features, the proposed method substantially reduces the computational cost of world model-based trajectory evaluation, achieving approximately 100× speedup while maintaining competitive planning performance.

Critical weaknesses
  1. In Table 3, although the pretrained projector improves EPDMS from 80.2 to 85.4, the resulting performance remains below several existing methods. Moreover, the compared methods use different input configurations and supervision settings (SV/MV, PF/PB), which directly causes that we cannot make any fair comparison to existing methods. While the authors present these methods as references, the only controlled baseline is a randomly initialized projector, which is insufficient to establish the advantages of the proposed world model pretraining over alternative pretraining strategies. Therefore, the downstream experiments provide limited evidence of the competitiveness of the learned representations.

  2. The paper lacks critical ablation studies on its core components, particularly the convolutional projector and SIGReg regularization. The authors do not investigate different feature compression ratios, simpler projector architectures, or the effect of removing SIGReg. Therefore, it is difficult to determine the individual contributions of these components to the reported performance and efficiency improvements. Furthermore, the experimental section is confusing and does not clearly distinguish the main comparisons from ablation studies. In particular, Table 2 mixes comparisons with existing world models and analyses of rollout training, training data scale, and the number of candidate trajectories, making the purpose and key findings of each experiment unclear.

Questions for authors
  1. In Table 3, the proposed method achieves 85.4 EPDMS, which remains below several existing methods. Could the authors further explain the performance gap and clarify the motivation for including methods with different input configurations and supervision settings (SV/MV, PF/PB)?

  2. Table 3 only compares the pretrained projector against random initialization. Could the authors provide comparisons with alternative pretraining strategies to better demonstrate the specific benefits of action-conditioned world model pretraining?

  3. Could the authors provide additional ablation studies on the core components, particularly the effect of removing SIGReg, varying feature compression ratios, and replacing the convolutional projector with simpler alternatives such as average pooling?

1 comment

We thank the Reviewer for the feedback. We have incorporated editions into our current draft and respond below.

Q1 & Q2 (and Weakness 1):

The primary contribution of our paper is JEPA-based world-model planning, while our downstream imitation-learning experiments demonstrate that the pretrained representations also benefit end-to-end driving without requiring future images at inference time. As mentioned in Section 4.4, existing imitation-learning methods are included in Table 3 for reference only, as they differ in experimental settings and some require industry-level computational resources, such as 32–64 A800 GPUs for several days. In contrast, our downstream experiments use only 2 A100 GPUs for 13 hours. The most direct controlled comparison is therefore between our pretrained projector and a randomly initialized projector under identical settings, as reported in Table 3, where EPDMS improves from 80.2 to 85.4.

Q3 (and Weakness 2)

We conducted additional ablation studies to evaluate different feature compression methods while reporting the shapes of the per-frame embeddings used for planning: (1) Learnable global projector, which adds additional layers on top of our learnable patch projector to further compress the patch embeddings into a single 256-dimensional vector per frame; (2) 2D average pooling (D=1024), which directly reduces the spatial dimensions of the DINO features while preserving the original embedding dimension of 1024; (3) 2D average pooling (D=256), which first reduces the spatial dimensions of the DINO features while preserving the original embedding dimension of 1024, then averages groups of four features to reduce the embedding dimension to 256; and (4) Learnable patch projector without SIGReg, which uses our learnable patch projector without SIGReg regularization.

Compression methodEPDMS ↑EPDMS† ↑FDE (m) ↓
Learnable global projector (256)63.086.06.7
2D average pooling (4, 8, 1024)67.388.74.9
2D average pooling (4, 8, 256)60.582.77.8
Learnable patch projector without SIGReg (4, 8, 256)52.283.96.9
Learnable patch projector (4, 8, 256)76.692.44.2

Ablation study of feature compression methods on 100 NAVSIMv2 navtest scenes. FDE is measured at the 4-second planning horizon. These results demonstrate that our learnable patch projector achieves the best driving performance and the lowest reported final displacement error, supporting the effectiveness of the proposed compression design.

We thank the reviewer for requesting channel and spatial compression ablations.

The existing DINO-WM/JEPA-WM models, with a planning embedding shape of (32, 16, 1024), can also be viewed as an ablation of spatial compression and dimensionality reduction relative to our proposed projector, which produces a compressed embedding shape of (4, 8, 256).

Due to time constraints, we trained each AD-E2E-JEPA projector variant for 5 epochs (navtrain, no rollout) to additionally compare channel and spatial ablations. Results:

Compression methodEPDMS ↑EPDMS† ↑FDE (m) ↓
Learnable patch projector, paper (4, 8, 256)56.083.87.2
Channel compression (4, 8, 64)61.984.29.3
Spatial compression (2, 4, 256)63.585.47.1
Channel + spatial compression (2, 4, 64)57.579.610.5

These preliminary results suggest that spatial compression and channel can further improve planning performance. Full training was incomplete at the rebuttal deadline; we plan to report the results in the camera-ready version if the paper is accepted.

Reviewer_u8jmOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents AD-E2E-JEPA, an action-conditioned world model that uses a learnable projector with SIGReg regularization to compress visual patch embeddings. It compares several JEPA-based world models on NAVSIM through goal-conditioned trajectory selection using ground-truth future images as goals, measuring planning performance and efficiency. It also examines whether the pretrained projector benefits downstream imitation learning through a separate transfer experiment.

Critical strengths

The proposed projector provides a simple and effective way to reduce the computational cost of world-model planning. By compressing both the spatial resolution and dimensionality of patch embeddings, it reduces planning time by approximately 100× compared with DINO-WM and JEPA-WM under the same 256-candidate setting, while maintaining comparable planning performance within the evaluated goal-conditioned protocol.

Critical weaknesses
  • The main experimental setting of the paper, zero-shot goal-conditioned planning, does not align with the requirements of real-world driving. It requires ground-truth future images, which are unavailable in actual driving. It does not demonstrate how AD-E2E-JEPA benefits real-world driving, nor is it sufficient for evaluating JEPA in realistic driving scenarios. The disconnect between the evaluation and the intended application substantially weakens the contribution.

  • The model in the imitation learning transfer setting does not require future images and could be applied to real-world driving. However, the authors do not compare it with other end-to-end models under the same settings, including input views, reliance on perception supervision, and the number of historical frames. Its performance also remains substantially below that of WA-JEPA. Therefore, these results do not convincingly demonstrate the proposed architecture’s contribution to improving real-world driving performance.

Questions for authors
  • Given that ground-truth future images are unavailable in actual driving, how should the goal-conditioned planning results be interpreted in terms of driving capability with JEPA?
  • Could the authors compare the proposed model with other end-to-end methods under the same settings under the imitation learning transfer protocol, including input views, perception supervision, and historical context?
  • What modifications were made to the original DINO-WM, JEPA-WM, and LeWM implementations, including their predictor architectures, parameter counts and rollout training settings?
  • How was the 8,192-trajectory vocabulary constructed, including the data source and clustering procedure?
1 comment

We thank the Reviewer for the feedback. We have incorporated editions into our current draft and respond below.

Q1 (and Weakness 1):

The motivation behind goal-conditioned zero-shot planning is to isolate the planning capability of the world model from policy learning, thereby avoiding the confounding effects of policy training. This evaluation protocol is widely adopted in JEPA-based world-model research to assess whether learned action-conditioned dynamics can support planning in unseen scenarios without training a task-specific policy [1][2][3]

We acknowledge that ground-truth future images are unavailable in real-world driving. Our goal-conditioned setting is therefore intended as a diagnostic evaluation of the world model's intrinsic planning capability, rather than a directly deployable driving framework. By evaluating the world model independently of policy learning, we aim to investigate whether JEPA-based representations can support trajectory selection for autonomous driving.

Q2 (and Weakness 2):

The primary contribution of our paper is JEPA-based world-model planning, while our downstream imitation-learning experiments demonstrate that the pretrained representations also benefit end-to-end driving without requiring future images at inference time. As mentioned in Section 4.4, existing imitation-learning methods are included in Table 3 for reference only, as they differ in experimental settings and some require industry-level computational resources, such as 32–64 A800 GPUs for several days. In contrast, our downstream experiments use only 2 A100 GPUs for 13 hours. The most direct controlled comparison is therefore between our pretrained projector and a randomly initialized projector under identical settings, as reported in Table 3, where EPDMS improves from 80.2 to 85.4.

Q3:

For a fair comparison, we use ViT-L-sized backbones across DINO-WM, JEPA-WM, and LeWM. DINO-WM and JEPA-WM use pretrained DINOv3 ViT-L encoders, whereas LeWM uses a ViT-L trained from scratch. The released implementations of DINO-WM and LeWM do not include rollout training. For our rollout variant, we compare the rollout-training procedure with JEPA-WM.

Q4:

The 8,192-trajectory vocabulary is constructed using K-means clustering on trajectories from the nuPlan dataset, following a standard practice in vocabulary-based end-to-end autonomous driving methods. We also emphasize that AD-E2E-JEPA is not restricted to clustering-based trajectory vocabularies and can accommodate other candidate sets, such as manually designed trajectories that span the trajectory space. We adopt the current vocabulary because clustering-based trajectory selection is widely used in candidate-based end-to-end driving methods.

[1] Zhou, Gaoyue, Hengkai Pan, Yann LeCun, and Lerrel Pinto. "Dino-wm: World models on pre-trained visual features enable zero-shot planning." arXiv preprint arXiv:2411.04983 (2024).

[2] Terver, Basile, Tsung-Yen Yang, Jean Ponce, Adrien Bardes, and Yann LeCun. "What drives success in physical planning with joint-embedding predictive world models?." arXiv preprint arXiv:2512.24497 (2025).

[3] Maes, Lucas, Quentin Le Lidec, Damien Scieur, Yann LeCun, and Randall Balestriero. "Leworldmodel: Stable end-to-end joint-embedding predictive architecture from pixels." arXiv preprint arXiv:2603.19312 (2026).

Reviewer_TyDxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes AD-E2E-JEPA, an action-conditioned world model for autonomous driving. It introduces a learnable projector with SIGReg regularization to compress visual representations for efficient latent-space planning. Experiments on NAVSIMv2 show approximately 100× faster planning than DINO-WM and JEPA-WM, while maintaining competitive performance. The pretrained projector also improves downstream imitation learning. However, the main architectural components lack sufficient ablation, and the use of ground-truth future images substantially limits the practical significance of the driving results. Stronger experimental evidence is needed to establish the method's contribution to autonomous driving.

Critical strengths

The proposed representation compression reduces planning time while maintaining reasonable trajectory accuracy.

The evaluation includes planning accuracy, efficiency, safety-related metrics, and downstream transfer, providing useful evidence of the method's potential.

Critical weaknesses

Insufficient ablation studies. The main contribution combines spatial compression, dimensionality reduction, and SIGReg regularization. However, the paper does not isolate their individual contributions. Comparisons with simple pooling, different compression ratios, and a projector without SIGReg are needed to justify the proposed design.

Limited evidence of real-time performance. Although planning is approximately 100× faster than the dense JEPA baselines, it still takes 0.8 seconds for 256 candidates and 4.7 seconds for 2,048 candidates. The paper does not evaluate end-to-end system latency or closed-loop driving performance.

Limited baseline evaluation. DINO-WM and JEPA-WM are evaluated on only 100 sampled scenes, while full-test comparisons are limited to LeWM. The paper also lacks uncertainty estimates for the relatively small performance differences on the 100-scene subset.

Questions for authors

Can the authors provide ablations isolating the effects of spatial compression, embedding dimensionality, and SIGReg regularization?

What is the complete inference latency, including visual encoding, and can the method support real-time closed-loop driving?

How stable are the results across different random subsets or training seeds?

2 comments

We thank the Reviewer for the feedback. We have incorporated editions into our current draft and respond below.

Q1 (and Weakness 1):

We conducted additional ablation studies to evaluate different feature compression methods while reporting the shapes of the per-frame embeddings used for planning: (1) Learnable global projector, which adds additional layers on top of our learnable patch projector to further compress the patch embeddings into a single 256-dimensional vector per frame; (2) 2D average pooling (D=1024), which directly reduces the spatial dimensions of the DINO features while preserving the original embedding dimension of 1024; (3) 2D average pooling (D=256), which first reduces the spatial dimensions of the DINO features while preserving the original embedding dimension of 1024, then averages groups of four features to reduce the embedding dimension to 256; and (4) Learnable patch projector without SIGReg, which uses our learnable patch projector without SIGReg regularization.

Compression methodEPDMS ↑EPDMS† ↑FDE (m) ↓
Learnable global projector (256)63.086.06.7
2D average pooling (4, 8, 1024)67.388.74.9
2D average pooling (4, 8, 256)60.582.77.8
Learnable patch projector without SIGReg (4, 8, 256)52.283.96.9
Learnable patch projector (4, 8, 256)76.692.44.2

Ablation study of feature compression methods on 100 NAVSIMv2 navtest scenes. FDE is measured at the 4-second planning horizon. These results demonstrate that our learnable patch projector achieves the best driving performance and the lowest reported final displacement error, supporting the effectiveness of the proposed compression design.

We thank the reviewer for requesting channel and spatial compression ablations.

The existing DINO-WM/JEPA-WM models, with a planning embedding shape of (32, 16, 1024), can also be viewed as an ablation of spatial compression and dimensionality reduction relative to our proposed projector, which produces a compressed embedding shape of (4, 8, 256).

Due to time constraints, we trained each AD-E2E-JEPA projector variant for 5 epochs (navtrain, no rollout) to additionally compare channel and spatial ablations. Results:

Compression methodEPDMS ↑EPDMS† ↑FDE (m) ↓
Learnable patch projector, paper (4, 8, 256)56.083.87.2
Channel compression (4, 8, 64)61.984.29.3
Spatial compression (2, 4, 256)63.585.47.1
Channel + spatial compression (2, 4, 64)57.579.610.5

These preliminary results suggest that spatial compression and channel can further improve planning performance. Full training was incomplete at the rebuttal deadline; we plan to report the results in the camera-ready version if the paper is accepted.

Q2 (and Weakness 2):.

The reported planning time already includes visual encoding. Specifically, we measure planning time from the completion of data loading to the selection of the trajectory with the lowest latent distance, including visual encoding and world-model rollouts. The average total evaluation time per scene is approximately 2.44–2.51 seconds, including additional overhead from environment initialization, library imports, model initialization and checkpoint loading, scene/image loading, preprocessing, driving metric computation, and result writing.

We also emphasize that the reported planning time of 0.8 seconds for 256 candidate trajectories is achieved without any inference acceleration techniques, such as pruning, quantization, or knowledge distillation. These techniques could potentially further reduce inference latency.

Weakness 3:

As shown in Table 2, DINO-WM and JEPA-WM are computationally expensive, requiring 91.8 and 101.0 seconds per scene, respectively. Evaluating these models on all 12,146 test scenes would therefore require approximately 13 and 14 days, respectively, on a single A100 GPU for trajectory selection alone, excluding data loading, preprocessing, and metric computing. Given these computational constraints, full-set evaluation is prohibitively expensive.

Since the 12,146 test scenes are sequentially indexed by driving sequence, sampling consecutive scenes could introduce temporal redundancy and limit coverage of the test set. Therefore, we systematically subsample 100 scenes at approximately uniform intervals across the entire test set. This strategy provides broad coverage of the test set while substantially reducing the computational cost of evaluating the baselines.

Q3

For training, the default random seed is 0. We conducted two additional experiments with AD-E2E-JEPA on the navtrain split without rollout training, using random seeds 42 and 666. We report their performance alongside the seed-0 results from the paper below:

SettingEPDMS ↑EPDMS† ↑FDE (m) ↓
AD-E2E-JEPA (training seed 0, paper)76.692.44.2
AD-E2E-JEPA (training seed 42)70.9290.794.38
AD-E2E-JEPA (training seed 666)77.3191.684.90
Mean ± std (three training seeds)74.94 ± 3.5091.62 ± 0.814.49 ± 0.36

For 100 sub-sampled testing scenes reported in the paper, since the 12,146 test scenes are sequentially indexed by driving sequence, sampling consecutive scenes could introduce temporal redundancy and limit coverage of the test set. Therefore, we systematically subsample 100 scenes at approximately uniform intervals across the entire test set. This strategy provides broad coverage of the test set while substantially reducing the computational cost of evaluating the baselines.

To assess the impact of evaluation-subset sampling, we additionally randomly sampled 100 test scenes from the entire testing dataset using each of three random seeds (0, 42, and 666). We compare the results with those reported in the paper below:

Evaluation settingEPDMS ↑EPDMS† ↑FDE (m) ↓
AD-E2E-JEPA (paper, 100 sampled scenes)76.692.44.2
AD-E2E-JEPA (paper, full test set)63.580.16.3
AD-E2E-JEPA (sampling seed 0)65.3685.947.01
AD-E2E-JEPA (sampling seed 42)68.1487.226.18
AD-E2E-JEPA (sampling seed 666)75.3989.804.89
Mean ± std (three sampling seeds)69.63 ± 5.1887.65 ± 1.976.03 ± 1.07

Overall, the results are generally stable across training seeds and evaluation subsets.

Reviewer_7Vd5OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

AD-E2E-JEPA adapts action-conditioned JEPA world models to goal-conditioned autonomous-driving planning. A learnable projector with SIGReg reduces the number and dimensionality of patch embeddings, allowing efficient evaluation of candidate trajectories against a future-image goal. Experiments on NAVSIMv2 report substantial planning speedups and show that the pretrained projector also improves downstream imitation learning.

Critical strengths
  1. The planning efficiency improvement is substantial. Table 2 reports a reduction from approximately 92–101 seconds to 0.8 seconds for evaluating 256 candidate trajectories, while retaining competitive planning performance under the same goal-conditioned protocol.

  2. The pretrained projector provides downstream benefits. Table 3 shows an improvement in imitation-learning EPDMS from 80.2 to 85.4 compared with a randomly initialized projector, supporting the usefulness of the learned representations beyond goal-conditioned planning.

Critical weaknesses
  1. The evaluation scope is limited. All experiments use NAVSIM, and comparisons with DINO-WM and JEPA-WM cover only 100 scenes. Although the proposed method is evaluated on all 12,146 test scenes, these stronger baselines are absent from the full-set comparison. Planning also assumes access to ground-truth future images as goals. This diagnostic setting does not establish generalization across datasets or performance with practically available goals.

  2. The proposed components are not sufficiently isolated. The paper does not separately evaluate spatial compression, channel reduction, and SIGReg, or compare the projector with simpler compression alternatives. Consequently, it remains unclear which design choices are necessary to preserve planning quality while reducing computation.

  3. Important performance trade-offs are insufficiently explained. In Table 2, rollout training on navtrain improves FDE from 4.2 to 3.5 m on the 100-scene subset, but reduces EPDMS from 76.6 to 70.4; Table 4 shows DAC falling from 87.0 to 77.0. Additionally, increasing training data from 10 to 70 hours without rollout training does not improve full-set EPDMS, while several training settings also change. The paper should better explain these outcomes and distinguish the effects of additional data, rollout training, and configuration changes.

  4. Qualitative evidence is too limited to explain the quantitative results. Figure 3 shows only one scene, with thin trajectory lines and no clear scale or error annotations. It does not illustrate difficult interactions or representative failures that could explain the gap between endpoint accuracy and driving safety. Clearer trajectory overlays and selected success and failure cases would make the analysis more informative.

Questions for authors
  1. How were the 100 comparison scenes selected, and how does their scenario composition compare with the full test set?
  2. Are existing controlled results available that separate spatial compression, channel reduction, and SIGReg, or compare against simple pooling?
  3. Which failure cases explain the reduction in drivable-area compliance after rollout training despite improved endpoint accuracy?
  4. How do the authors interpret the limited benefit of additional training data without rollout training, given the simultaneous changes in training configuration?
4 comments

We thank the Reviewer for the feedback. We have incorporated editions into our current draft and respond below.

Q1 (and Weakness 1):

As shown in Table 2, DINO-WM and JEPA-WM are computationally expensive, requiring 91.8 and 101.0 seconds per scene, respectively. Evaluating these models on all 12,146 test scenes would therefore require approximately 13 and 14 days, respectively, on a single A100 GPU for trajectory selection alone, excluding data loading, preprocessing, and metric computing. Given these computational constraints, full-set evaluation is prohibitively expensive.

Since the 12,146 test scenes are sequentially indexed by driving sequence, sampling consecutive scenes could introduce temporal redundancy and limit coverage of the test set. Therefore, we systematically subsample 100 scenes at approximately uniform intervals across the entire test set. This strategy provides broad coverage of the test set while substantially reducing the computational cost of evaluating the baselines.

We acknowledge that ground-truth future images are unavailable in real-world driving. Our goal-conditioned setting is therefore intended as a diagnostic evaluation of the world model's intrinsic planning capability, rather than a directly deployable driving framework. By evaluating the world model independently of policy learning, we aim to investigate whether JEPA-based representations can support trajectory selection for autonomous driving.

Q2 (and Weakness 2):

We conducted additional ablation studies to evaluate different feature compression methods while reporting the shapes of the per-frame embeddings used for planning: (1) Learnable global projector, which adds additional layers on top of our learnable patch projector to further compress the patch embeddings into a single 256-dimensional vector per frame; (2) 2D average pooling (D=1024), which directly reduces the spatial dimensions of the DINO features while preserving the original embedding dimension of 1024; (3) 2D average pooling (D=256), which first reduces the spatial dimensions of the DINO features while preserving the original embedding dimension of 1024, then averages groups of four features to reduce the embedding dimension to 256; and (4) Learnable patch projector without SIGReg, which uses our learnable patch projector without SIGReg regularization.

Compression methodEPDMS ↑EPDMS† ↑FDE (m) ↓
Learnable global projector (256)63.086.06.7
2D average pooling (4, 8, 1024)67.388.74.9
2D average pooling (4, 8, 256)60.582.77.8
Learnable patch projector without SIGReg (4, 8, 256)52.283.96.9
Learnable patch projector (4, 8, 256)76.692.44.2

Ablation study of feature compression methods on 100 NAVSIMv2 navtest scenes. FDE is measured at the 4-second planning horizon. These results demonstrate that our learnable patch projector achieves the best driving performance and the lowest reported final displacement error, supporting the effectiveness of the proposed compression design.

We thank the reviewer for requesting channel and spatial compression ablations. Due to time constraints, we trained each AD-E2E-JEPA projector variant for 5 epochs (navtrain, no rollout, 100 testing scenes) for comparison. Results:

Compression methodEPDMS ↑EPDMS† ↑FDE (m) ↓
Learnable patch projector, paper (4, 8, 256)56.083.87.2
Channel compression (4, 8, 64)61.984.29.3
Spatial compression (2, 4, 256)63.585.47.1
Channel + spatial compression (2, 4, 64)57.579.610.5

These preliminary results suggest that spatial compression and channel can further improve planning performance. Full training was incomplete at the rebuttal deadline; we plan to report the results in the camera-ready version if the paper is accepted.

Q3 (and Weakness 3):

Currently, goal-conditioned zero-shot planning selects a trajectory based on latent distance rather than directly optimizing for each specific driving metric; therefore, rollout training could improve overall driving performance, but improvements in a specific driving metric (e.g., drivable-area compliance) are not guaranteed. For AD-E2E-JEPA, a more statistically robust evaluation is on the full set of test scenes in Table 2; in this case, rollout training generally helps improve performance.

Q4

We thank the reviewer for raising this concern. When training on the trainval split, we used 4 GPUs without extensive hyperparameter tuning. We investigated and found that the best checkpoint without rollout training occurred at epoch 13 out of 30, whereas the best checkpoints for other settings were obtained near the end of training. This suggests differences in training dynamics, and further hyperparameter tuning may potentially improve performance on the trainval split without rollout training.

Reviewer_TyDxOpenReview ↗

Response to Authors

Thank you for the response. My main concern remains the lack of ablation studies. Results that separate the effects of spatial compression, dimensionality reduction and SIGReg have not been provided yet. I would need to see these results to judge whether the proposed design is justified.

We thank the reviewer for the response. We have just added the latest ablation study results to the main rebuttal thread: (1) learnable global projector, (2) 2D average pooling (D=1024), (3) 2D average pooling (D=256), and (4) learnable patch projector without SIGReg.

The existing DINO-WM/JEPA-WM models, with a planning embedding shape of (16, 32, 1024), can also be viewed as an ablation of spatial compression and dimensionality reduction relative to our proposed projector, which produces a compressed embedding shape of (4, 8, 256).

Additional small-scale experiments (trained for up to 5 epochs) are still running and will be compared against our proposed projector (also trained for 5 epochs to ensure a fair comparison). We will post these results before the rebuttal deadline.

Additional small-scale experiments (up to 5 training epochs) have been completed, and the results have been added to the main rebuttal thread.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.