- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces ST-JEPA, a self-supervised learning method for end-to-end motion planning in autonomous driving. ST-JEPA builds upon the joint embedding predictive architecture (JEPA, Assran et al., 2023) to learn scene, trajectory and video representations that improve downstream planning. Scene-trajectory representations are learned early in the encoder. This enables more attention to visual representations than in related work, which introduces trajectory queries only after the encoder. The downstream motion planning head is adapted from the DrivoR (Kirby et al., 2026) model.
Critical strengths
- ST-JEPA combines two complementary self-supervision objectives. The scene-trajectory JEPA objective combined with a standard video-based JEPA objectives need no human annotations and lead to SOTA performance on NAVSIM and HUGSIM.
- The paper includes detailed ablation studies that cover the cumulative effect of all contributions (see Table 3) and different information fusion approaches (see Table 7). This highlights significant engineering effort and the improvements compared to the DrivoR baseline model.
Critical weaknesses
-
The evaluation using NAVSIM and HUGSIM is limited to nominal driving conditions. Measuring differences between pseudo-simulation on NAVSIM and closed-loop simulation on HUGSIM is valuable, yet limited compared to measuring generalization from nominal to long-tail driving scenarios.
-
ST-JEPA uses a learned temporal position embeddings, which prevents generalization to longer temporal sequences than the ones seen during training.
Questions for authors
These questions are clarification questions and suggestions for the rebuttal:
-
Does ST-JEPA generalize to unusual, long-tail driving scenarios? E.g. perform a small-scale cross-dataset ablation with a zero-/ few-shot evaluation of models (pre)trained on nuScenes/NAVSIM on WOD-E2E (Xu et al., 2026) or KITScenes LongTail (Wagner et al., 2026).
-
Does using RoPE (Su et al., 2024) as temporal embedding enable generalization to longer temporal sequences? For example, perform a small-scale ablation with longer planning horizons than used during training.
Xu et al., 2026: WOD-E2E: Waymo Open Dataset for End-to-End Driving in Challenging Long-tail Scenarios. In CVPR.
Wagner et al., 2026: Reasoning models do not yet follow their reasoning in autonomous driving: The KITScenes LongTail Dataset. In arXiv.
Su et al., 2024: Roformer: Enhanced transformer with rotary position embedding. In Neurocomputing.
3 comments
Thank you for your response. The additional experiments improve the work by showing generalization w.r.t. dataset characteristics and longer prediction horizons. The cross-dataset experiment would be even more convincing by reporting pseudo-simulation/ composite metrics like RFS (Xu et al., 2026) or MMS (Wagner et al., 2026) in addition to displacement errors. Overall, I’m leaning toward increasing my rating to 4.
Additional RFS Results for Cross-Dataset Evaluation
Dear Reviewer o9Ne,
Thank you very much for your thoughtful follow-up and for recognizing the value of our additional experiments.
Following your suggestion, we have further extended the zero-shot cross-dataset evaluation by including RFS (Xu et al., 2026) alongside the displacement metrics.
Table R1: Zero-shot cross-dataset evaluation on WOD-E2E val.
| Model | ADE (m) ↓ | FDE (m) ↓ | RFS ↑ |
|---|---|---|---|
| ST-JEPA | 2.7 | 6.1 | 7.5 |
| DrivoR | 3.1 | 6.8 | 6.9 |
As shown in Table R1, ST-JEPA outperforms DrivoR by 0.6 RFS points, further supporting its cross-dataset generalization beyond displacement errors.
We sincerely appreciate your constructive suggestion, thoughtful reconsideration, and positive assessment of our work. Thank you again for your time and valuable feedback.
We sincerely thank the reviewer for the careful evaluation and for recognizing our comprehensive ablations and performance improvements.
Response 1: Long-Tail Scenario Generalization
Response:
To assess long-tail generalization, we provide (1) zero-shot cross-dataset evaluation on WOD-E2E and (2) difficulty-stratified closed-loop evaluation on HUGSIM.
(1) Zero-Shot Cross-Dataset Evaluation on WOD-E2E.
We conduct a zero-shot evaluation on WOD-E2E val, which contains challenging long-tail driving scenarios. Both ST-JEPA and DrivoR are evaluated under identical settings, using four historical frames sampled at approximately 0.5-second intervals. Since our evaluation configuration uses three front-facing cameras, we zero-fill the rear-camera input for both models.
Table R1: Zero-shot cross-dataset evaluation on WOD-E2E val.
ST-JEPA reduces ADE by 0.4 m (12.9%) and FDE by 0.7 m (10.3%) compared with DrivoR, demonstrating improved trajectory prediction accuracy under zero-shot cross-dataset transfer.
(2) Difficulty-Stratified Closed-Loop Evaluation on HUGSIM.
To further assess generalization under challenging closed-loop conditions, we report a difficulty-stratified evaluation on 345 HUGSIM scenarios, using the same evaluator and evaluation settings for both methods.
Table R2: HUGSIM closed-loop performance across scenario difficulty levels. All values are percentages; higher is better.
ST-JEPA outperforms the reproduced DrivoR baseline on both RC and HD across all four difficulty levels. On hard scenarios, ST-JEPA improves RC by 4.99 points and HD by 7.72 points, indicating stronger closed-loop performance under challenging driving conditions.
We further report the evaluator's no-at-fault-collision (NC) and time-to-collision (TTC) scores to examine collision-related safety. These are normalized evaluator scores rather than raw collision counts, with higher values indicating better safety performance.
Table R3: Collision-related safety metrics on HUGSIM across difficulty levels. All values are percentages; higher is better.
ST-JEPA achieves substantial NC/TTC improvements on hard (12.59/14.31 points) and extreme (8.22/10.15 points) scenarios, demonstrating improved safety under challenging driving conditions. Across independent training runs, the standard deviations of RC-Avg and HD-Avg remain within 0.5 percentage points for both models, supporting the stability of the results.
Overall, the WOD-E2E results demonstrate improved zero-shot cross-dataset trajectory prediction, while the HUGSIM results provide complementary evidence of stronger closed-loop performance and improved safety on challenging long-tail scenarios. Together, these evaluations support ST-JEPA's generalization beyond nominal driving conditions.
Response 2: Temporal RoPE and Longer-History Generalization
Response:
To investigate temporal generalization, we replace the learned temporal position embeddings with RoPE and evaluate ST-JEPA with longer historical observations. All experiments are conducted using the ST-JEPA variant without the video branch to isolate the effect of temporal encoding.
Table R4: Temporal encoding and longer-history generalization. All experiments use ST-JEPA without the video branch.
As shown above, RoPE achieves comparable performance to learned temporal embeddings under the original 4-frame setting (93.57 vs. 93.55 PDMS). Moreover, when extending the evaluation history from 4 to 6 frames without retraining, RoPE further improves PDMS to 93.64. These results suggest that RoPE enables ST-JEPA to accommodate longer temporal observations while maintaining stable planning performance.