acceptodds
Withdrawn from ICLR 2027

Auto-JEPA: A Latent World Model of Continuous Intent for End-to-End Autonomous Driving

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

12 views

Abstract

Existing autonomous-driving world models typically perform dense prediction of future videos, occupancy states, BEV representations, or agent motion. We argue that planning need not reconstruct the complete future world, but only focus on scene features that affect future ego action. Based on this perspective, we propose Auto-JEPA, an action-oriented latent world model that learns continuous future driving intent through joint-embedding prediction. Given visual observations, ego-motion history, and navigation commands, Auto-JEPA predicts an intent embedding aligned with the latent representation of the future ego trajectory. The predicted intent retrieves executable trajectories from a fixed trajectory memory, which are then ranked by a scene-conditioned candidate selection module. Auto-JEPA keeps the visual encoder frozen, requires no explicit perception annotations, and uses no learned trajectory generator. By optimizing only task-specific modules for trajectory representation, intent prediction, and candidate selection, Auto-JEPA achieves 91.3 PDMS on NAVSIM v1 and 89.1 EPDMS on NAVSIM v2. Semantic occlusion experiments show that masking dynamic-agent regions induces an average intent change 2.97× that of equal-area random masking. Moreover, occluding vehicles that affect future driving substantially changes the predicted intent and selected trajectory, whereas both remain essentially unchanged when non-influential vehicles are occluded. These results show that future-intent prediction encourages the model to focus on planning-relevant visual features and supports high-quality planning without dense future-world modeling.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Autonomous Driving, World Models, Joint-Embedding Predictive Architecture, Latent Intent Prediction, Trajectory Planning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ZEPCOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents Auto-JEPA, a driving planner that predicts a latent representation of the future ego trajectory from visual observations, ego history, and navigation commands. The predicted representation retrieves candidates from a fixed trajectory memory, followed by scene scoring and drivable area filtering. Experiments cover NAVSIM v1 and v2, with additional component ablations and visual occlusion analyses.

Critical strengths
  1. The retrieval formulation provides a clear alternative to learned trajectory generation. A shared trajectory embedding connects visual prediction with executable candidate geometry, and the complete system achieves competitive performance using only a front camera.

  2. The component ablations help explain the planning pipeline. The paper examines intent prediction, candidate scoring, feasibility gating, and retrieval pool size. The controlled occlusion analysis also provides evidence that predictions depend on dynamic agent information.

Critical weaknesses
  1. The benefit of the proposed latent prediction formulation is not sufficiently isolated. The fixed medoid baseline removes scene conditioning entirely, rather than comparing alternative prediction objectives. Comparisons with direct waypoint prediction followed by retrieval, or direct candidate ranking using the same backbone, memory, and scorer, are needed to establish the specific contribution of joint embedding prediction. Predicting only the ego trajectory also leaves the claimed distinction from a conventional imitation planner unclear.

  2. The evaluation does not establish a consistent performance or efficiency advantage. Under the original NAVSIM v2 evaluator, Auto-JEPA scores 85.6, below several reported baselines; its updated score of 89.1 is not directly comparable with their original scores. The paper also lacks reactive closed loop evaluation and computational measurements, despite using a 24 layer predictor, a large trajectory memory, and separate selection modules.

Questions for authors
  1. What capability makes Auto-JEPA a world model beyond a planner trained to predict an embedding of the demonstrated ego trajectory?

  2. How does latent prediction compare with direct waypoint prediction followed by retrieval, and with direct candidate ranking, using the same visual backbone, memory, scorer, and training budget?

  3. Could the authors compare against the CLOVER generator using the same scorer and candidate budget, and evaluate strong baselines under the same NAVSIM v2 implementation?

  4. How are the 110,335 memory trajectories selected, and are all entries restricted to training data? How do memory size and coverage affect retrieval quality and performance on rare maneuvers?

  5. What are the total parameter count, memory footprint, and batch size one latency, including visual encoding, retrieval, scoring, and gating?

Reviewer_Jfs7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Auto-JEPA, a latent trajectory prediction and retrieval framework designed for end-to-end autonomous driving. It first establishes a frozen target representation space using a trajectory autoencoder. Then, based on front-facing camera observations, the ego-vehicle’s motion history, and navigation commands, it predicts continuous future trajectory representations through feature alignment, cosine alignment, and contrastive learning. During inference, 300 candidate trajectories are retrieved from a fixed trajectory library, followed by a selection process based on scene scoring and feasibility gating. Auto-JEPA achieves a PDMS of 91.3 on NAVSIM v1, and EPDMS scores of 85.6 and 89.1 under the original and updated NAVSIM v2 evaluation protocols, respectively.

Critical strengths
  • The framework features a coherent design with a clear motivation. It effectively delineates the roles of intent prediction, trajectory retrieval, quality scoring, and feasibility filtering, offering a practical approach to motion planning that avoids the need for dense future scene reconstruction.
  • The results on NAVSIM v1 are competitive. Using only a single front-facing camera, the model achieves a PDMS of 91.3 across the full set of 12,146 navtest scenarios; even without the scene scorer, the retrieval-plus-gating mechanism still yields a score of 87.6.
Critical weaknesses
  • The contributions of the core methodology are not sufficiently disentangled. The use of fixed medoid-based contrastive learning demonstrates the importance of scene-aware retrieval, but it does not prove that JEPA-based representations are superior to alternatives such as coordinate-based retrieval, trajectory classification, or simple motion-feature-based approaches.
  • The rationale for retrieval within the latent space lacks validation. High-quality trajectory reconstruction does not guarantee that cosine distance is an appropriate metric for planning-oriented retrieval. Furthermore, the paper lacks ablation studies on the individual training loss components.
  • The evidence regarding occlusion primarily highlights sensitivity to specific instances; the current comparisons and examples are insufficient to demonstrate that the model accurately identifies decision-relevant vehicles or generates appropriate planning responses.
Questions for authors
  • Could you provide a comparison between coordinate-based retrieval and trajectory-vocabulary classification, keeping the backbone, memory bank, number of candidates, and scorer constant? Additionally, could you include a baseline that relies solely on historical motion and navigation commands, along with an ablation study of the training losses?
  • Does the latent cosine distance correspond to meaningful trajectory differences? Could you report the best performance—both for querying trajectory representations and among the Top-K candidates—to disentangle prediction, coverage, and selection errors?
  • Is it possible to define vehicle relevance independently, incorporate occlusion controls based on positional matching, and provide additional metrics such as end-to-end latency, memory overhead, and uncertainty estimates for the results?
  • Could you include test results from other closed-loop evaluation datasets to demonstrate the generalizability of the proposed method?
Reviewer_BpSsOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Auto-JEPA predicts a continuous representation of future ego motion instead of reconstructing future scene observations. A trajectory autoencoder first defines a frozen target embedding space. A transformer then predicts eight trajectory-latent tokens from frozen V-JEPA 2 visual features, ego-motion history, and navigation commands, using alignment and contrastive losses. At inference, the predicted latent retrieves 300 candidates from a memory of 110,335 recorded trajectories. A separately trained scene-conditioned scorer and drivable-area gate select the final plan. The system reports 91.3 PDMS on NAVSIM v1 and 89.1 EPDMS under the updated NAVSIM v2 evaluator. Component ablations and occlusion experiments examine the contributions of retrieval, scoring, gating, and visual information.

Critical strengths
  1. The retrieval formulation is clear and empirically promising. The method separates motion representation, candidate retrieval, and scene-specific selection in a way that is easy to inspect. Table 3 shows that intent-based retrieval with the gate already reaches 87.6 PDMS, while the scorer raises this to 91.3; the improvement is not hidden inside an unexplained final module. The equal-area random-mask control also gives useful evidence that the predicted representation responds more strongly to dynamic-agent regions than to arbitrary image removal.
Critical weaknesses
  1. The ablations do not isolate the benefit of the JEPA representation. Replacing the predicted intent with a fixed medoid removes scene-conditioned retrieval altogether, so the drop to 52.6 PDMS is unsurprising. The paper does not compare against direct trajectory prediction followed by geometric nearest-neighbor retrieval, or another scene-conditioned embedding under the same memory and selection modules. Without those controls, the results support conditional retrieval, but leave unclear whether the learned target space and joint-embedding losses are the important ingredients.

  2. The limits and cost of the trajectory memory remain underexplored. The planner can only return trajectories already stored in its fixed memory. Varying the retrieved candidate count does not establish whether that memory covers unusual maneuvers or unfamiliar road geometry. The paper also calls the approach lightweight without reporting end-to-end latency or memory use, despite searching 110,335 embeddings and scoring 300 candidates. Oracle coverage and runtime measurements are needed to judge the practical tradeoff against learned trajectory generation.

Questions for authors
  1. Could you compare latent retrieval with geometric retrieval from directly predicted waypoints, keeping the visual backbone, trajectory memory, scorer, gate, and candidate budget matched? Please also separate the effects of trajectory-space pretraining, cosine alignment, and InfoNCE.

  2. How do oracle best-candidate quality, achieved PDMS, latency, and memory use change as the trajectory bank shrinks or encounters held-out maneuver types? Reporting the gap between the best trajectory in the full bank, the retrieved set, and the selected output would distinguish coverage, retrieval, and scoring failures.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.