acceptodds
Under review as a conference paper at ICLR 2027

ST-JEPA: JEPA-Enhanced Temporal Scene–Trajectory Representation Learning for End-to-End Autonomous Driving

Abstract

End-to-end autonomous driving requires compact representations that preserve planning-relevant spatial and temporal context. A common design in recent ViT-based E2E planners is to introduce trajectory queries only after visual feature extraction, which limits the interaction between planning intent and visual representation learning during encoding. Meanwhile, in compact-token variants, scene tokens receive only auxiliary supervision without direct self-supervision on their latent representations. To address these limitations, we present **ST-JEPA**, a temporal scene–trajectory representation learning framework with two key designs: scene–trajectory encoding and JEPA-enhanced representation learning. First, ST-JEPA injects camera-aware scene tokens and ego-conditioned trajectory tokens directly into the visual encoder, enabling them to interact with image patches throughout feature extraction. Interleaved temporal global-register attention further aggregates cross-frame context and propagates the updated representations through subsequent backbone blocks. Second, ST-JEPA incorporates joint-embedding predictive learning through two complementary mechanisms. An online scene–trajectory JEPA objective predicts compact scene and trajectory representations from masked observations, using targets produced by an EMA teacher from unmasked inputs. This training-only objective provides latent self-supervision directly at the compact planning bottleneck. In parallel, we adapt JEPA pretraining to driving videos and integrate the resulting spatiotemporal latent features into the encoder by projecting them into the token space and fusing them with the native scene representation. ST-JEPA achieves state-of-the-art performance on NAVSIM-v1, NAVSIM-v2, and the closed-loop HUGSIM benchmark, with systematic ablations validating the effectiveness of the proposed scene–trajectory encoding, temporal interaction, and JEPA-based representation learning. The code will be made available upon publication.

Primary area:
applications to robotics, autonomy, planning
Keywords:
End-to-End Autonomous Driving, Joint-Embedding Predictive Architecture (JEPA)

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_o9NeOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces ST-JEPA, a self-supervised learning method for end-to-end motion planning in autonomous driving. ST-JEPA builds upon the joint embedding predictive architecture (JEPA, Assran et al., 2023) to learn scene, trajectory and video representations that improve downstream planning. Scene-trajectory representations are learned early in the encoder. This enables more attention to visual representations than in related work, which introduces trajectory queries only after the encoder. The downstream motion planning head is adapted from the DrivoR (Kirby et al., 2026) model.

Critical strengths
  1. ST-JEPA combines two complementary self-supervision objectives. The scene-trajectory JEPA objective combined with a standard video-based JEPA objectives need no human annotations and lead to SOTA performance on NAVSIM and HUGSIM.
  2. The paper includes detailed ablation studies that cover the cumulative effect of all contributions (see Table 3) and different information fusion approaches (see Table 7). This highlights significant engineering effort and the improvements compared to the DrivoR baseline model.
Critical weaknesses
  1. The evaluation using NAVSIM and HUGSIM is limited to nominal driving conditions. Measuring differences between pseudo-simulation on NAVSIM and closed-loop simulation on HUGSIM is valuable, yet limited compared to measuring generalization from nominal to long-tail driving scenarios.

  2. ST-JEPA uses a learned temporal position embeddings, which prevents generalization to longer temporal sequences than the ones seen during training.

Questions for authors

These questions are clarification questions and suggestions for the rebuttal:

  1. Does ST-JEPA generalize to unusual, long-tail driving scenarios? E.g. perform a small-scale cross-dataset ablation with a zero-/ few-shot evaluation of models (pre)trained on nuScenes/NAVSIM on WOD-E2E (Xu et al., 2026) or KITScenes LongTail (Wagner et al., 2026).

  2. Does using RoPE (Su et al., 2024) as temporal embedding enable generalization to longer temporal sequences? For example, perform a small-scale ablation with longer planning horizons than used during training.

Xu et al., 2026: WOD-E2E: Waymo Open Dataset for End-to-End Driving in Challenging Long-tail Scenarios. In CVPR.

Wagner et al., 2026: Reasoning models do not yet follow their reasoning in autonomous driving: The KITScenes LongTail Dataset. In arXiv.

Su et al., 2024: Roformer: Enhanced transformer with rotary position embedding. In Neurocomputing.

3 comments

We sincerely thank the reviewer for the careful evaluation and for recognizing our comprehensive ablations and performance improvements.

Response 1: Long-Tail Scenario Generalization

Reviewer's Comment (Weakness 1 / Question 1): Does ST-JEPA generalize to challenging long-tail driving scenarios beyond nominal conditions?

Response:

To assess long-tail generalization, we provide (1) zero-shot cross-dataset evaluation on WOD-E2E and (2) difficulty-stratified closed-loop evaluation on HUGSIM.

(1) Zero-Shot Cross-Dataset Evaluation on WOD-E2E.

We conduct a zero-shot evaluation on WOD-E2E val, which contains challenging long-tail driving scenarios. Both ST-JEPA and DrivoR are evaluated under identical settings, using four historical frames sampled at approximately 0.5-second intervals. Since our evaluation configuration uses three front-facing cameras, we zero-fill the rear-camera input for both models.

Table R1: Zero-shot cross-dataset evaluation on WOD-E2E val.

ModelADE (m) ↓FDE (m) ↓
ST-JEPA2.76.1
DrivoR3.16.8

ST-JEPA reduces ADE by 0.4 m (12.9%) and FDE by 0.7 m (10.3%) compared with DrivoR, demonstrating improved trajectory prediction accuracy under zero-shot cross-dataset transfer.

(2) Difficulty-Stratified Closed-Loop Evaluation on HUGSIM.

To further assess generalization under challenging closed-loop conditions, we report a difficulty-stratified evaluation on 345 HUGSIM scenarios, using the same evaluator and evaluation settings for both methods.

Table R2: HUGSIM closed-loop performance across scenario difficulty levels. All values are percentages; higher is better.

HUGSIM Difficulty# ScenariosDrivoR RC / HDST-JEPA RC / HD
Easy4878.46 / 71.1779.85 / 74.17
Medium12755.26 / 50.7358.49 / 55.10
Hard8630.36 / 18.9035.35 / 26.62
Extreme8440.49 / 26.4242.82 / 27.31
Overall34548.68 / 39.7251.88 / 43.89

ST-JEPA outperforms the reproduced DrivoR baseline on both RC and HD across all four difficulty levels. On hard scenarios, ST-JEPA improves RC by 4.99 points and HD by 7.72 points, indicating stronger closed-loop performance under challenging driving conditions.

We further report the evaluator's no-at-fault-collision (NC) and time-to-collision (TTC) scores to examine collision-related safety. These are normalized evaluator scores rather than raw collision counts, with higher values indicating better safety performance.

Table R3: Collision-related safety metrics on HUGSIM across difficulty levels. All values are percentages; higher is better.

HUGSIM DifficultyDrivoR NC / TTCST-JEPA NC / TTC
Hard45.31 / 37.1557.90 / 51.46
Extreme45.14 / 41.5653.36 / 51.71

ST-JEPA achieves substantial NC/TTC improvements on hard (12.59/14.31 points) and extreme (8.22/10.15 points) scenarios, demonstrating improved safety under challenging driving conditions. Across independent training runs, the standard deviations of RC-Avg and HD-Avg remain within 0.5 percentage points for both models, supporting the stability of the results.

Overall, the WOD-E2E results demonstrate improved zero-shot cross-dataset trajectory prediction, while the HUGSIM results provide complementary evidence of stronger closed-loop performance and improved safety on challenging long-tail scenarios. Together, these evaluations support ST-JEPA's generalization beyond nominal driving conditions.

Response 2: Temporal RoPE and Longer-History Generalization

Reviewer's Comment (Weakness 2 / Question 2): Does using RoPE as temporal embedding enable generalization to longer temporal sequences?

Response:

To investigate temporal generalization, we replace the learned temporal position embeddings with RoPE and evaluate ST-JEPA with longer historical observations. All experiments are conducted using the ST-JEPA variant without the video branch to isolate the effect of temporal encoding.

Table R4: Temporal encoding and longer-history generalization. All experiments use ST-JEPA without the video branch.

Temporal EncodingTraining HistoryEvaluation HistoryPDMS ↑Relative to Learned 4-frame
Learned embedding4 frames4 frames93.550.00
Temporal RoPE4 frames4 frames93.57+0.02
Temporal RoPE4 frames6 frames93.64+0.09

As shown above, RoPE achieves comparable performance to learned temporal embeddings under the original 4-frame setting (93.57 vs. 93.55 PDMS). Moreover, when extending the evaluation history from 4 to 6 frames without retraining, RoPE further improves PDMS to 93.64. These results suggest that RoPE enables ST-JEPA to accommodate longer temporal observations while maintaining stable planning performance.

Thank you for your response. The additional experiments improve the work by showing generalization w.r.t. dataset characteristics and longer prediction horizons. The cross-dataset experiment would be even more convincing by reporting pseudo-simulation/ composite metrics like RFS (Xu et al., 2026) or MMS (Wagner et al., 2026) in addition to displacement errors. Overall, I’m leaning toward increasing my rating to 4.

AuthorsOpenReview ↗

Additional RFS Results for Cross-Dataset Evaluation

Dear Reviewer o9Ne,

Thank you very much for your thoughtful follow-up and for recognizing the value of our additional experiments.

Following your suggestion, we have further extended the zero-shot cross-dataset evaluation by including RFS (Xu et al., 2026) alongside the displacement metrics.

Table R1: Zero-shot cross-dataset evaluation on WOD-E2E val.

ModelADE (m) ↓FDE (m) ↓RFS ↑
ST-JEPA2.76.17.5
DrivoR3.16.86.9

As shown in Table R1, ST-JEPA outperforms DrivoR by 0.6 RFS points, further supporting its cross-dataset generalization beyond displacement errors.

We sincerely appreciate your constructive suggestion, thoughtful reconsideration, and positive assessment of our work. Thank you again for your time and valuable feedback.

Reviewer_UKtqOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ST-JEPA extends DrivoR with ego-conditioned trajectory tokens and temporal attention inside the visual backbone. A JEPA objective supervises the compact scene and trajectory tokens during training. At inference, a frozen driving-adapted V-JEPA 2 encoder supplies additional features through learned projection and aggregation modules. The decoder and scorer architectures remain unchanged. Evaluation covers NAVSIM-v1, NAVSIM-v2 and zero-shot transfer to HUGSIM.

Critical strengths
  1. The method makes a coherent change to DrivoR's representation. Ego-conditioned trajectory tokens participate in visual encoding and temporal interaction, and the JEPA objective supervises the compact representation consumed by the planner. These choices connect the auxiliary objective directly to planning. The decoder and scorer architectures are retained. The teacher and predictor are removed at inference, while the video encoder remains active.

  2. The empirical evaluation includes meaningful controls and tests beyond NAVSIM-v1. The full model gains 2.10 PDMS over reproduced DrivoR in the matched setting, and the variant without the video branch also improves. This supports a contribution from the task-adapted representation itself. Alternative fusion schemes, patch targets and same-size encoder controls are already included. Results on NAVSIM-v2 and zero-shot HUGSIM show that the reported gains extend beyond NAVSIM-v1, although those comparisons are less tightly controlled than the internal ablations.

Critical weaknesses
  1. The methodological increment over prior work is modest. DrivoR already uses ego-conditioned trajectory queries in its decoder. Register-based cross-frame interaction followed by further visual processing has precedent in VGGT Omega, and driving-video JEPA adaptation has been explored in Drive-JEPA. ST-JEPA's specific contribution lies mainly in moving the queries into visual encoding and jointly supervising compact scene and trajectory tokens, together with the prompt and feature-integration design. The controlled experiments support this complete configuration, including improvements without the video branch. The trajectory-token comparison changes both where ego conditioning enters and how many prefix tokens the encoder processes, so it supports the combined design without isolating the effect of earlier conditioning. The compact target improves over the patch target by 0.18 PDMS, and the fusion alternatives are also separated by only a few tenths of a point. No variation across independent training runs is reported for these comparisons. The study establishes a useful extension of DrivoR more clearly than a robust design principle for planning representations. In my assessment, these useful but incremental changes do not provide sufficient additional insight to meet the acceptance threshold.

  2. Important parts of the training procedure remain insufficiently specified. For driving-video adaptation, the appendix gives the clip format, frozen encoder status and separation from supervised OpenScene trainval data, but omits the video sources, splits, scale and adaptation budget. The same-size encoder controls are useful, yet these omissions leave the additional resources behind the gain unclear. For the online JEPA path, Equation 12 specifies joint planning and JEPA training, but the encoder-side prompt is described in terms of pretraining. Its use during this joint stage and the mapping from low-resolution spatial features to compact scene and trajectory token slots remain unclear. These details affect interpretation of the prompt ablation.

Questions for authors
  1. Which driving-video datasets and splits were used, at what scale, and for how many adaptation updates? The appendix describes the clip format and data separation but leaves the pretraining resources unclear.

  2. How are the low-resolution image features mapped to the scene and trajectory token slots, and is the encoder-side prompt used during the joint training in Equation 12 or only in a separate pretraining stage?

  3. How many independent training runs support the target and fusion comparisons, and how large is their variation relative to the reported gains? This matters particularly to the fine-grained design recommendations.

  4. I could not identify a matching public navtest record for the NAVSIM-v1 results. Were these results obtained locally or through the official server, and which evaluator revision was used?

4 comments

We sincerely thank the reviewer for the careful evaluation and for recognizing our coherent representation design, meaningful ablations, and performance improvements.

Response 1: Methodological Novelty and Contributions

Reviewer's Comment (Weakness 1): The methodological novelty is limited, and the current ablations do not isolate the effect of early ego conditioning from increased token capacity.

Response:

ST-JEPA has two complementary contributions.

1. Ego-conditioned spatiotemporal scene–trajectory representation. Unlike DrivoR, which introduces trajectory queries in the decoder, ST-JEPA injects camera-aware scene tokens and ego-conditioned trajectory tokens before visual encoding. These tokens interact with image patches and historical observations inside the backbone, allowing planning conditions to shape spatiotemporal representation formation while maintaining a compact planner interface.

2. JEPA-enhanced compact-token learning. ST-JEPA applies online JEPA supervision directly to planner-facing scene–trajectory tokens using an EMA teacher, rather than supervising only patches or generic video features. A low-resolution visual prompt provides complementary spatial context, while frozen driving-adapted video features further enhance the compact planning representation.

Together, these contributions establish a planning-conditioned compact representation enhanced by predictive latent supervision, rather than a simple combination of independent modules.

Relation to prior work. We do not claim trajectory queries, register-based temporal interaction, or driving-video JEPA pretraining as individually new. Compared with DrivoR, we introduce ego conditioning during visual representation formation rather than only in the decoder. Unlike VGGT Omega, which employs register-based interactions for geometric reconstruction, ST-JEPA uses planning-conditioned tokens with direct JEPA supervision for autonomous driving. Unlike Drive-JEPA, which focuses on driving-video representation learning, ST-JEPA additionally supervises task-specific planning tokens through a separate online JEPA objective and integrates frozen video features into the planner.

Capacity-matched ablations. We acknowledge that the original trajectory-token comparison changed both the conditioning location and token count. We therefore provide additional controls:

Table R1: Capacity-matched ablations of prefix-token design and ego-conditioning location.

VariantBackbone Prefix TokensEgo-conditioning LocationVideoPDMS
Temporal scene-token referenceKsK_s sceneDecoder onlyOff93.23
Scene-token capacity control(Ks+Kq)(K_s+K_q) sceneDecoder onlyOff93.25
Capacity-matched late conditioningKsK_s scene + KqK_q trajectoryAfter visual/temporal encodingOff93.45
Early ST-JEPAKsK_s scene + KqK_q trajectoryBefore visual encodingOff93.55

The three capacity-matched variants share the same total number of backbone prefix tokens (Ks+KqK_s+K_q), token dimension, backbone, temporal modules, decoder, scorer, training data, and optimization schedule, with the video branch disabled. Early and late conditioning also share the same ego projection and decoder-side ego features, differing only in the ego-conditioning injection point.

Increasing generic scene tokens yields only +0.02 PDMS (93.23 → 93.25). In contrast, introducing structured scene–trajectory tokens improves performance by +0.20 PDMS (93.25 → 93.45), while incorporating ego conditioning during visual encoding provides an additional +0.10 PDMS gain (93.45 → 93.55). These results support the overall scene–trajectory representation design, rather than attributing the improvement solely to increased token capacity or the conditioning injection location.

Additional evidence. Without the video branch, our online JEPA design improves PDMS from 93.55 to 94.08. Notably, compact scene–trajectory-token JEPA outperforms patch-level JEPA (93.88 vs. 93.70 PDMS), supporting the effectiveness of directly supervising planner-facing representations.

Summary. These results support our two core contributions: Ego-conditioned spatiotemporal scene–trajectory representation and JEPA-enhanced compact planning tokens. The capacity-matched controls demonstrate the benefits of jointly modeling scene and trajectory representations beyond simply increasing token capacity, while the JEPA ablations support the effectiveness of predictive supervision on planner-facing representations. We will further clarify these design principles and their distinctions from prior work in the revised manuscript.

Response 2: Driving-Video Pretraining Details

Reviewer's Comment (Weakness 2 / Question 1): The driving-video adaptation lacks details on datasets, data splits, training scale, and computational budget.

Response:

We provide the detailed driving-video adaptation settings below and will include them in the supplementary material. The adaptation uses only the OpenScene training split, with NAVSIM evaluation data excluded.

Table R2: Driving-video pretraining configuration and computational budget.

ItemSpecification
Dataset and splitOpenScene train split (test split excluded)
Data scale1,310 clips (~100.42 hours)
Input configurationFront camera, 8 frames at 2 FPS, 256 × 512
Encoder initializationPublic V-JEPA 2 ViT-L checkpoint
Adaptation budget100 epochs, 30,000 configured updates, global batch size 64
Computational resources8 NVIDIA H20 GPUs, ~15 hours

Response 3: Low-Resolution Prompt Design

Reviewer's Comment (Weakness 2 / Question 2): How are low-resolution features mapped to scene and trajectory tokens, and is the encoder-side prompt used during the joint training in Equation 12 or only in a separate pretraining stage?

Response:

In the JEPA mode, the low-resolution feature is not converted into or appended as a scene/trajectory token. For each masked image patch, the encoder replaces its original feature with learnable mask token + corresponding low-resolution feature, rather than the mask token alone. Scene and trajectory tokens are constructed separately and processed jointly with these patches by the visual-temporal backbone, so the prompt affects compact tokens only indirectly.

This operation is part of the joint objective L=Lplan+λJEPALJEPAL = L_{\mathrm{plan}} + \lambda_{\mathrm{JEPA}} L_{\mathrm{JEPA}} (Eq. 12), while the EMA teacher receives the unmasked input. At inference, masking is disabled, so the prompt and JEPA teacher/predictor are not executed.

Response 4: Statistical Reliability of Ablations

Reviewer's Comment (Weakness 1 / Question 3): The target and fusion improvements are small, and their stability across independent training runs remains unclear.

Response:

We agree that these fine-grained comparisons should be supported by multiple independent runs. We therefore repeat the key target and fusion ablations with five independent random seeds under identical training and evaluation settings.

Table R3: Statistical reliability of target and temporal-fusion ablations across five independent seeds.

VariantOriginal single-run PDMSNumber of seedsMean PDMSStd.BestWorst
Patch-level feature target (prompt off)93.70593.710.0393.7593.67
Compact scene/trajectory-token target (prompt off)93.88593.900.0393.9493.86
Post-backbone concatenation93.01593.000.0293.0392.97
In-backbone global-register attention93.23593.240.0293.2793.21

Across five independent runs, compact-token targets outperform patch-level targets by 0.19 PDMS on average, while in-backbone temporal fusion improves over post-backbone concatenation by 0.24 PDMS. These improvements exceed the observed run-to-run standard deviations (0.02–0.03 PDMS), supporting their stability under the evaluated settings.

Response 5: NAVSIM Evaluation Protocol

Reviewer's Comment (Question 4): Were the NAVSIM-v1 results evaluated locally or on the official server, and which evaluator revision was used?

Response:

We have evaluated ST-JEPA on the official NAVSIM-v1 navtest leaderboard under the NAVSIM v1.1 evaluation protocol, achieving 95.37 PDMS, consistent with the result reported in Table 1.

Reviewer_UKtqOpenReview ↗

Updated Assessment Following the Author Response

I thank the authors for addressing my concerns. The capacity-matched ablation in Table R1 isolates the effect of introducing ego conditioning earlier in the encoder and shows a modest gain over late conditioning. The five-seed results in Table R3 carry more weight in my assessment. They provide consistent evidence for two design choices in this planner: supervising compact scene and trajectory tokens and integrating temporal information within the visual backbone.

I continue to view ST-JEPA as an incremental advance over DrivoR. With the additional controls and repeated runs, however, I find the empirical contribution sufficient to warrant acceptance. I am therefore revising my overall rating from 2 to 3, weak acceptance.

Dear Reviewer UKtq,

Thank you very much for revisiting our rebuttal and for your thoughtful assessment. We greatly appreciate your recognition of the empirical contributions of ST-JEPA. We are particularly grateful for your careful reconsideration of our work in light of the additional evidence provided in our rebuttal.

Thank you again for your time, consideration, and valuable feedback.

Reviewer_G99XOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ST-JEPA, a compact-token end-to-end driving planner built on the DrivoR planning interface. The central motivation is that existing ViT-based planners often introduce planning/trajectory queries only after visual encoding, and that compact scene tokens are typically trained only through downstream planning losses.

ST-JEPA makes two primary modifications:

Temporal scene–trajectory encoding. Camera-aware scene tokens and ego-conditioned trajectory tokens are injected as prefix tokens into the visual backbone, allowing them to interact with image patches throughout visual encoding. The model performs per-camera temporal fusion through global-register attention at selected backbone depths over a four-frame history.

JEPA-enhanced representation learning. The paper adds:

a training-only online JEPA objective that predicts EMA-teacher scene and trajectory token representations from masked visual observations; and a frozen driving-adapted V-JEPA2 branch whose compressed spatiotemporal features are fused into the planner’s scene memory at inference. The downstream DrivoR trajectory decoder and scorer remain unchanged. The paper reports 94.85/95.37 PDMS on NAVSIM-v1 under the 85k/103k training regimes, 91.8 EPDMS on NAVSIM-v2, and zero-shot HUGSIM results of 51.9 RC-Avg and 43.9 HD-Avg. The ablation reports a cumulative improvement from 92.75 to 94.85 PDMS under the matched NAVSIM-v1 85k setting.

Critical strengths

The paper targets a relevant limitation of compact-token planners: visual tokens, planning intent, and temporal context are frequently coupled only late in the architecture. Injecting ego-conditioned trajectory tokens inside the encoder is a reasonable way to test whether planning-conditioned visual representations can be more effective than post-hoc planning queries.

The use of scene and trajectory prefix tokens, followed by in-backbone temporal interaction, is conceptually clean. In particular, updating tokens across time and then propagating them through later visual blocks is better motivated than merely concatenating independently encoded frame features at the planner input.

Predicting scene/trajectory token representations rather than reconstructing pixels or dense patch embeddings is a sensible design decision. It directly regularizes the representations that the planner consumes, while the EMA teacher and predictor are removed at inference.

The temporal-fusion comparison is particularly useful: in-backbone global-register attention reaches 93.23 PDMS, compared with 93.01 for post-backbone concatenation and 92.90 for learnable-query fusion. The token-level JEPA target also improves over the patch-level alternative in the reported setting.

The reported NAVSIM-v1 and NAVSIM-v2 results are competitive with the listed methods. The HUGSIM experiment is also valuable because it tests transfer beyond the planner’s primary benchmark and shows a sizable reported improvement over DrivoR.

The authors explicitly state that cross-paper benchmark rows may differ in implementation, evaluation version, supervision, sensors, and training setup, while the reproduced DrivoR baseline and the paper’s ablations use a shared pipeline. This is good experimental practice.

The authors acknowledge that the driving-video JEPA branch only processes front-camera history, leaving side/rear temporal information outside this pretrained representation. This is an important limitation rather than a minor implementation detail.

Critical weaknesses
  1. The paper combines several individually plausible ideas: planning-conditioned tokens inside a visual backbone, temporal attention over compact tokens, EMA-teacher latent prediction, driving-video JEPA pretraining, and query-based feature compression. The resulting system is effective, but it is less clear which conceptual advance is fundamentally new relative to compact-token driving planners, temporal visual encoders, and prior JEPA-for-driving approaches.

The manuscript should articulate more precisely whether its primary novelty is:

putting trajectory-conditioned tokens inside visual encoding; conducting temporal fusion inside the encoder rather than afterward; applying JEPA directly to compact scene/trajectory planning tokens; or combining online token JEPA with frozen driving-video JEPA features. At present, all four are presented as central, but the evidence is stronger for the usefulness of the full combination than for a distinctive new principle that separates ST-JEPA from a careful integration of known components.

  1. The full model adds a frozen driving-adapted V-JEPA2 ViT-L branch, learned multi-scale projections, a query aggregation module, and 128 additional scene tokens to the planner memory. The reported gain from the video JEPA branch is substantial: from 94.08 to 94.85 PDMS. However, this comparison changes not only the pretraining signal but also inference-time architecture, token capacity, backbone scale, and likely compute.

The paper compares against “Standard ViT-L” and “Generic V-JEPA ViT-L,” which is helpful, but the comparison remains incomplete without a capacity- and compute-matched baseline. For example, a strong control would use the same ViT-L/video branch, the same 128 aggregated tokens, and the same planner-side integration, but with image pretraining, generic video pretraining, random/frozen features, or driving-video pretraining without the JEPA formulation. This would better isolate the source of the reported gain.

3.The paper is transparent that Table 1 and Table 2 include source-reported reference values, but the headline state-of-the-art claims rely heavily on those comparisons. Since training data, auxiliary supervision, sensor inputs, model sizes, and evaluator implementations may differ, the comparisons should be interpreted cautiously.

For an ICLR-level empirical claim, the paper would be stronger with more controlled comparisons to the closest relevant baselines—especially Drive-JEPA and a temporal version of DrivoR—using a matched backbone, temporal input length, training data, and evaluation pipeline. At minimum, the paper should more prominently distinguish controlled gains over reproduced DrivoR from broader leaderboard comparisons.

  1. The paper states that driving-video JEPA pretraining clips are separate from the OpenScene supervised planner data. However, the precise dataset provenance, scale, camera configuration, filtering, and overlap policy with NAVSIM-v1/v2 and HUGSIM should be specified more rigorously.

In particular:

Which driving-video dataset(s) are used for V-JEPA2 adaptation? Is there any scene-, route-, log-, or geographically overlapping data with downstream evaluation sets? What is the amount of video pretraining relative to planner training? Are all baseline methods allowed comparable external driving-video pretraining? How much of the final gain comes from domain-specific pretraining versus a larger auxiliary backbone? These details are essential to interpret the reported generalization and benchmark improvements.

  1. The HUGSIM result is encouraging, but it is limited to an aggregate result over 345 scenarios. For a method emphasizing temporal representation and planning safety, the evaluation would be substantially stronger with:

per-scenario-type breakdowns, e.g., turns, vulnerable road users, dense traffic, intersection negotiation, occlusion, and sudden lead-vehicle behavior; collision/failure categories, not only RC-Avg and HD-Avg; variance across seeds or confidence intervals; qualitative failure cases where temporal fusion or JEPA features lead to unsafe or overly conservative behavior; a comparison against an equally temporal, capacity-matched baseline. The aggregate HUGSIM gain is not yet enough to establish why the learned representation improves closed-loop behavior.

  1. The method adds temporal processing over four frames, in-backbone global-register attention, and an inference-time frozen V-JEPA2 ViT-L branch. Yet the paper does not provide an end-to-end latency, FLOPs, parameter count, memory footprint, or hardware-specific throughput comparison.

This omission matters because the paper motivates compact representations for efficient planning. The V-JEPA2 branch operates at a lower resolution and only on the front camera, but it still introduces an additional large video encoder. The paper should quantify the inference overhead relative to DrivoR and explain whether the approach can meet real-time driving constraints.

  1. The paper demonstrates that in-backbone temporal fusion is preferable to two post-backbone fusion options, but several important questions remain:

How does performance vary with temporal horizon, frame rate, and the number of temporal fusion layers? Is per-camera temporal fusion preferable to cross-camera-and-time fusion under equivalent capacity? How sensitive is the method to camera dropout, temporal corruption, frame misalignment, or ego-state noise? Does the trajectory-token mechanism help uniformly across route-following, braking, turning, and interaction-heavy scenarios? What is the effect of the JEPA loss weight, masking ratio, teacher momentum, and prompt-token coverage? Does the online JEPA objective improve representation quality independently of the driving-video branch on a closed-loop metric? The presented ablations establish usefulness, but not robustness or sensitivity.

Questions for authors

Please clarify the exact provenance and scale of the driving-video data used to adapt V-JEPA2. What safeguards ensure no overlap with NAVSIM-v1/v2 evaluation data or HUGSIM scenarios?

Can the authors provide wall-clock latency, FLOPs, parameter counts, and peak memory for DrivoR, the temporal scene–trajectory encoder, and the full ST-JEPA model? What hardware and batch size are used?

Can the authors add a capacity-matched control for the driving-video branch: identical ViT-L architecture, query aggregation, and 128 128 added scene tokens, but without driving-video JEPA adaptation?

How does ST-JEPA perform when temporal history is shortened or corrupted? In particular, what happens under camera dropout, missing frames, timestamp jitter, or noisy ego-motion estimates?

Does the HUGSIM improvement hold consistently across scenario classes? Please provide per-category RC/HD scores and representative failure cases.

Why is the driving-video JEPA branch restricted to front-camera inputs while the main planner uses multiple cameras? Have the authors tested a multi-camera version, or a front-camera-only planner baseline to separate the auxiliary branch’s coverage mismatch from its representation benefit?

Can the authors clarify whether validation PDMS was used for extensive hyperparameter/model selection, and how many variants were evaluated before selecting the final checkpoint reported on navtest?

6 comments

We sincerely thank the reviewer for the careful evaluation and insightful suggestions, which help improve the quality of our work.

Response 1: Novelty and Core Contributions

Reviewer's Comment (Weakness 1): The conceptual novelty of ST-JEPA remains unclear, as the method combines existing techniques without sufficiently distinguishing its core contribution.

ST-JEPA has two complementary contributions.

1. Ego-conditioned spatiotemporal scene–trajectory representation. Unlike DrivoR, which introduces trajectory queries in the decoder, ST-JEPA injects camera-aware scene tokens and ego-conditioned trajectory tokens before visual encoding. These tokens interact with image patches and historical observations inside the backbone, allowing planning conditions to shape spatiotemporal representation formation while maintaining a compact planner interface.

2. JEPA-enhanced compact-token learning. ST-JEPA applies online JEPA supervision directly to planner-facing scene–trajectory tokens using an EMA teacher, rather than supervising only patches or generic video features. A low-resolution visual prompt provides complementary spatial context, while frozen driving-adapted video features further enhance the compact planning representation.

Together, these contributions establish a planning-conditioned compact representation enhanced by predictive latent supervision, rather than a simple combination of independent modules.

Relation to prior work. We do not claim trajectory queries, register-based temporal interaction, or driving-video JEPA pretraining as individually new. Compared with DrivoR, we introduce ego conditioning during visual representation formation rather than only in the decoder. Unlike VGGT Omega, which employs register-based interactions for geometric reconstruction, ST-JEPA uses planning-conditioned tokens with direct JEPA supervision for autonomous driving. Unlike Drive-JEPA, which focuses on driving-video representation learning, ST-JEPA additionally supervises task-specific planning tokens through a separate online JEPA objective and integrates frozen video features into the planner.

Capacity-matched ablations. We acknowledge that the original trajectory-token comparison changed both the conditioning location and token count. We therefore provide additional controls:

Table R1: Capacity-matched ablations of prefix-token design and ego-conditioning location.

VariantBackbone Prefix TokensEgo-conditioning LocationVideoPDMS
Temporal scene-token referenceKsK_s sceneDecoder onlyOff93.23
Scene-token capacity control(Ks+Kq)(K_s+K_q) sceneDecoder onlyOff93.25
Capacity-matched late conditioningKsK_s scene + KqK_q trajectoryAfter visual/temporal encodingOff93.45
Early ST-JEPAKsK_s scene + KqK_q trajectoryBefore visual encodingOff93.55

The three capacity-matched variants share the same total number of backbone prefix tokens (Ks+KqK_s+K_q), token dimension, backbone, temporal modules, decoder, scorer, training data, and optimization schedule, with the video branch disabled. Early and late conditioning also share the same ego projection and decoder-side ego features, differing only in the ego-conditioning injection point.

Increasing generic scene tokens yields only +0.02 PDMS (93.23 → 93.25). In contrast, introducing structured scene–trajectory tokens improves performance by +0.20 PDMS (93.25 → 93.45), while incorporating ego conditioning during visual encoding provides an additional +0.10 PDMS gain (93.45 → 93.55). These results support the overall scene–trajectory representation design, rather than attributing the improvement solely to increased token capacity or the conditioning injection location.

Additional evidence. Without the video branch, our online JEPA design improves PDMS from 93.55 to 94.08. Notably, compact scene–trajectory-token JEPA outperforms patch-level JEPA (93.88 vs. 93.70 PDMS), supporting the effectiveness of directly supervising planner-facing representations.

Summary. These results support our two core contributions: Ego-conditioned spatiotemporal scene–trajectory representation and JEPA-enhanced compact planning tokens. The capacity-matched controls demonstrate the benefits of jointly modeling scene and trajectory representations beyond simply increasing token capacity, while the JEPA ablations support the effectiveness of predictive supervision on planner-facing representations. We will further clarify these design principles and their distinctions from prior work in the revised manuscript.

Response 2: Capacity-Matched Comparison of Video JEPA

Reviewer’s Comment (Weakness 2 / Question 3): The gain from driving-video JEPA may stem from increased model capacity and computation rather than pretraining itself. Controlled comparisons under matched architectures are needed to isolate the contribution of driving-video JEPA adaptation.

Response:

We agree that the contribution of driving-video JEPA adaptation should be distinguished from increased model capacity. We therefore include a capacity-comparable image-pretrained ViT-L baseline and conduct controlled comparisons among V-JEPA variants using identical architectures, input configurations, multi-scale projections, query aggregation modules, 128 additional scene tokens, and planner-side integration.

Table R2: Ablation of video feature sources and pretraining strategies.

Video BranchPretrainingPDMS
None (ST-JEPA w/o Video JEPA)—94.08
Standard ViT-LImage pretraining94.13
Generic V-JEPA ViT-LGeneric video pretraining94.15
Driving-video V-JEPA2 ViT-LDriving-video adaptation94.65
Driving-video V-JEPA2 + low-resolution promptDriving-video adaptation + prompt94.85

All variants are evaluated under the same 85k training and evaluation protocol, with identical decoder, scorer, and optimization settings. Compared with the no-video reference (94.08 PDMS), the capacity-comparable Standard ViT-L and generic V-JEPA achieve 94.13 and 94.15 PDMS, respectively, yielding only marginal improvements (+0.05/+0.07 PDMS). In contrast, driving-video V-JEPA2 achieves 94.65 PDMS, outperforming the architecture-matched generic V-JEPA by +0.50 PDMS. This indicates that the improvement cannot be attributed solely to increased model capacity. Incorporating the low-resolution prompt during driving-video pretraining further improves performance to 94.85 PDMS (+0.20), without additional inference computation.

These results support the benefit of driving-domain JEPA adaptation under matched V-JEPA architectures and planner-side integration, beyond simply adding an auxiliary video branch.

Response 3: Fairness of Baseline Comparisons

Reviewer’s Comment (Weakness 3): The SOTA comparisons involve potentially different training data, model architectures, and evaluation settings. More controlled comparisons with closely related methods, particularly Drive-JEPA and temporal DrivoR, are needed.

Response:

We agree that source-reported leaderboard results should be distinguished from controlled comparisons. To address this concern, we highlight our comparisons with reproduced DrivoR and its temporal variant under the same training data, visual backbone, decoder, scorer, optimization schedule, and evaluation pipeline.

Table R3: Controlled comparisons with reproduced DrivoR and temporal variants.

ModelInput FramesPDMS
Reproduced DrivoR192.75
Temporal DrivoR493.23
ST-JEPA w/o Online JEPA & Video JEPA493.55
ST-JEPA w/o Video JEPA494.08
Full ST-JEPA494.85

Video JEPA refers to the auxiliary frozen driving-adapted V-JEPA2 feature branch.

Under the matched four-frame setting, full ST-JEPA improves upon Temporal DrivoR by +1.62 PDMS (93.23 → 94.85). Specifically, introducing scene–trajectory encoding improves performance to 93.55 PDMS (+0.32), while incorporating online JEPA supervision and the low-resolution pretraining prompt further increases performance to 94.08 PDMS (+0.53). The additional driving-adapted Video JEPA branch further improves performance to 94.85 PDMS (+0.77). These results demonstrate improvements beyond simply introducing temporal inputs into DrivoR.

Regarding Drive-JEPA, differences in planning architecture and training formulation prevent a strictly architecture-matched comparison. We therefore treat its source-reported results as contextual references rather than direct evidence of individual component gains. We will revise the manuscript to emphasize controlled improvements over reproduced DrivoR and its temporal variants, while clearly distinguishing broader leaderboard comparisons.

Response 4: Driving-Video Pretraining Data and Potential Overlap

Reviewer's Comment (Weakness 4 / Question 1): The provenance, scale, and configuration of the driving-video pretraining data remain insufficiently specified, and potential overlap with NAVSIM and HUGSIM evaluation sets needs to be ruled out.

Response:

We clarify the data provenance, split policy, and configuration of driving-video JEPA adaptation. The adaptation uses only the OpenScene training split, excluding the held-out test split used for NAVSIM-v1/v2 evaluation. HUGSIM is a separate simulation benchmark and is not included in the video pretraining data. Therefore, no held-out evaluation samples from NAVSIM-v1/v2 or HUGSIM are used for driving-video adaptation.

Table R4: Driving-video JEPA adaptation data and configuration.

ItemSpecification
Dataset and splitOpenScene train split (test split excluded)
Data scale1,310 clips (~100.42 hours)
Camera configurationFront camera only
Video sampling8 frames at 2 FPS
Input resolution256 × 512
Encoder initializationPublic V-JEPA 2 ViT-L checkpoint

Response 5: HUGSIM Closed-Loop Evaluation

Reviewer’s Comment (Weakness 5 / Question 5): The aggregate HUGSIM results do not adequately demonstrate consistent improvements across driving scenarios. Per-category performance, failure-case analysis, and statistical reliability are needed.

Response:

To assess generalization across challenging closed-loop driving scenarios, we evaluate ST-JEPA and reproduced DrivoR on all 345 HUGSIM scenarios, reporting difficulty-stratified results under identical evaluation settings.

Table R5: HUGSIM closed-loop performance across scenario difficulty levels. All values are percentages; higher is better.

HUGSIM Difficulty# ScenariosDrivoR RC / HDST-JEPA RC / HD
Easy4878.46 / 71.1779.85 / 74.17
Medium12755.26 / 50.7358.49 / 55.10
Hard8630.36 / 18.9035.35 / 26.62
Extreme8440.49 / 26.4242.82 / 27.31
Overall34548.68 / 39.7251.88 / 43.89

ST-JEPA outperforms the reproduced DrivoR baseline on both RC and HD across all four difficulty levels. On hard scenarios, ST-JEPA improves RC by 4.99 points and HD by 7.72 points, indicating stronger closed-loop performance under challenging driving conditions.

We further report the evaluator's no-at-fault-collision (NC) and time-to-collision (TTC) scores to examine collision-related safety. These are normalized evaluator scores rather than raw collision counts, with higher values indicating better safety performance.

Table R6: Collision-related safety metrics on HUGSIM across difficulty levels. All values are percentages; higher is better.

HUGSIM DifficultyDrivoR NC / TTCST-JEPA NC / TTC
Hard45.31 / 37.1557.90 / 51.46
Extreme45.14 / 41.5653.36 / 51.71

ST-JEPA achieves substantial NC/TTC improvements on hard (12.59/14.31 points) and extreme (8.22/10.15 points) scenarios, demonstrating improved safety under challenging driving conditions. Across independent training runs, the standard deviations of RC-Avg and HD-Avg remain within 0.5 pp for both models, supporting the stability of the results.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal and Reviewer Feedback

Dear Area Chair,

We sincerely thank the reviewers and the Area Chair for their constructive feedback. We are encouraged that two reviewers have expressed positive assessments of our work following the rebuttal, recognizing the additional experimental evidence and our efforts to address their concerns.

During the rebuttal, we conducted extensive additional experiments to address the reviewers' main concerns:

  1. Novelty and controlled comparisons: Clarified our core contributions and provided ablations to validate the benefits of early ego conditioning, compact-token JEPA supervision, and driving-video adaptation.
  2. Statistical reliability: Conducted five-seed experiments confirming consistent improvements with low variance.
  3. Generalization and safety: Added zero-shot WOD-E2E evaluation, difficulty-stratified HUGSIM results, and collision-related safety metrics.
  4. Robustness and efficiency: Evaluated temporal generalization, sensor perturbations, and computational overhead.
  5. Reproducibility: Clarified pretraining data, implementation details, and the official NAVSIM evaluation protocol.

We believe these additional results have substantially addressed the reviewers' major concerns and strengthened the empirical evidence supporting ST-JEPA.

We sincerely appreciate the reviewers' constructive feedback and positive reconsideration. We respectfully hope the Area Chair will take these additional results and positive developments into account when making the final decision.

Thank you very much for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.