acceptodds
Under review as a conference paper at ICLR 2027

Learning to Drive from the Future, Planning Trajectories in One Forward Pass

Abstract

World–action models (WAMs) enrich autonomous-driving policies with future-state supervision, yet it remains unclear which visual representations best support joint future-state and action learning and how to retain the benefits of future supervision in a single-pass planner. We introduce JEWAM, a Joint-Embedding World–Action Model for joint future-state and action learning, and use this framework to evaluate different visual representations. Comparisons of features from ViT, MAE, CroCo v2, DINOv3, and V-JEPA under a shared interface show that representation choice substantially affects planning. V-JEPA's predictive video features achieve the strongest planning results among the evaluated representations, motivating their adoption in the final model. We further show that future-state supervision improves action learning while its benefits are retained with single-pass, action-only inference. Direct full-horizon prediction outperforms the tested iterative variants at lower inference cost, supporting effective world–action learning without future rollout or iterative denoising at deployment. Using only front-view camera observations, JEWAM achieves 90.98 EPDMS on NAVSIM v2 NavTest and 39.86 on NavHard.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Autonomous Driving, World–Action Models, Visual Representation Learning

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.25 over 4 reviews, above 96% of the others.
Reviewer_nuGtOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

JEWAM is a joint-embedding world-action model. A student encoder reads four history frames. An EMA teacher with momentum 0.99925 and stop-gradient reads the full clip and provides future-state targets at the future token positions. A Mixture-of-Transformers predictor with a latent expert and an action expert predicts future features with an L1 loss and the full eight-step BEV trajectory with an L1 loss. The attention mask is asymmetric, following Metis. Future-state queries can read action keys and values. Action queries cannot read future-state keys and values. This makes the action path independent. At inference the teacher, the future queries, and the future head are removed. The full trajectory is produced in one forward pass, with no rollout and no denoising. The empirical core compares five encoders under a shared interface and a shared training budget: ViT, MAE, CroCo v2, DINOv3, and V-JEPA 2.1. Each encoder provides both the history context and the future targets. V-JEPA wins on navhard with 38.55 against 34.43 for DINOv3, so the final model uses V-JEPA, at ViT-G scale for the headline numbers. Results: 90.98 EPDMS on NAVSIM v2 navtest, 39.86 on navhard, and 92.08 PDMS on NAVSIM v1, all using a single front-view camera. The paper also compares the feed-forward predictor against two DiT variants and reports 29.10 Hz, which is 7.74 times the faster DiT at its best 15-step setting.

Critical strengths

The first strength is the representation study. This subfield needs this kind of controlled comparison. Many papers take a position on whether video-generative, latent-predictive, or image-pretrained features are better for driving, and most of them change several settings at the same time. Here the interface and the budget are fixed and only the encoder changes. The navhard results are 28.23 for ViT, 33.19 for MAE, 31.87 for CroCo v2, 34.43 for DINOv3, and 38.55 for V-JEPA. Predictive video features are worth about four points over the best image-pretrained alternative, and the gap is much larger on navhard than on navtest. I expect this table to be cited.

The second strength is the finding about inference. Future supervision helps during training, where navhard improves by 6.67 (33.19 becomes 39.86). At inference, removing the future tokens gives 39.78 against 39.86. Action-only inference is slightly better. This is a stronger statement than saying the benefit is retained. Combined with 29 Hz, the practical conclusion is clear: the world model is needed during training and is not needed during deployment.

I also want to note that the paper is direct about its own confounds. It states that the encoder comparison changes context and targets together. It states that the DiT variants change the training objective, so they do not isolate iterative sampling. This makes the rest of the claims easier to trust.

Critical weaknesses

The problem is that the paper names these confounds and does not remove them. In the representation study, changing the encoder changes the context and the targets at the same time. When V-JEPA wins, I cannot tell whether the history features are better, whether the future targets are better, or whether both are better. A third confound also exists: V-JEPA uses two-frame tubelets, and the other four encoders use one-frame tokens, so the temporal tokenisation changes too. A 2x2 crossing of context encoder against target encoder would answer this question, and it looks like a straightforward experiment. Without it, the central empirical claim of the paper is a correlation.

The future-supervision ablation has the same problem. The paper removes the future tokens, the teacher targets, and the future-state loss together, and it says so. This also removes an expert and its tokens, so the model has less capacity during training. Part of the 6.67 gain could be a capacity effect. A control with a target that carries no temporal information, for example predicting the current frame features or shuffled future frames at matched parameter count, would separate temporal information from model capacity.

The efficiency comparison is the third weakness. It is entirely internal. The DiT variants are built by the authors, trained from scratch for the same 20k steps, and flow matching usually needs its own tuning. The best DiT variant reaches 38.05 against 39.86 for the feed-forward model. The gap is real but modest. I read this as one implementation beating another implementation at a fixed budget. That is worth reporting, and it is not the same as showing that iterative prediction is worse in general. There is no latency or FLOPs comparison against any published diffusion planner. The frequency is measured on an Alibaba Zhenwu PPU-ZW810E. Almost no other paper reports on this hardware, so I cannot compare 29.10 Hz with published numbers. A800 or H100 timing, or FLOPs, would fix this.

One more point, which does not change the ranking. JEWAM wins on EPDMS because of the safety submetrics: NC 99.20, DAC 98.10, TTC 98.80, DDC 99.70. The comfort submetric goes the other way. On navtest, EC is 83.50, which is second lowest in Table 1. Latent-WAM is 87.30 and DriveWAM is 88.90. EP is 87.10, which is middling. The authors discuss the EC drop for navhard stage 2 and do not discuss the navtest EC value, which is the larger regression relative to the field. I would like a comment on whether this is a real trade-off or an artifact. The single front camera is also presented only as an efficiency advantage. It is one, and it is also a limitation for deployment. The paper should say this.

Minor: Fast-WAM (Yuan et al., 2026) asks almost exactly this paper's question, whether world-action models need test-time future imagination. It is cited in the related work and never appears in a comparison table. As with most submissions in this area, there are no seeds and no error bars.

Questions for authors

Can you run the context-encoder against target-encoder crossing, even with only V-JEPA and DINOv3 in both slots? Can you also match the temporal tokenisation across encoders?

Can you add a capacity control to the future-supervision ablation, using a non-informative future target at matched parameter count?

Can you report A800 or H100 latency, or FLOPs, and compare against at least one published iterative planner on the same hardware?

Can you comment on the navtest EC value of 83.50?

Can you add a comparison against Fast-WAM?

2 comments

We thank the reviewer for the careful review and high standards; these comments substantially help improve the rigor of our argument. We address each point below.

1. Confounding factors in the representation study and the 2×2 crossing (Weakness 1 / Question 1)

The reviewer is correct that the representation study simultaneously changes the context encoder, the target encoder, and the temporal tokenization. Following your suggestion, we conducted a 2×2 crossing of context encoder × target encoder, together with a no-future-supervision control. Results (NavHard EPDMS) are:

History encoder (student)No future supervisionDINOv3 future targetV-JEPA future target
DINOv3-300M29.8434.4333.79
V-JEPA-300M31.4535.0438.55

The table can be read in three directions:

(i) Encoder quality contributes, but is not the main factor. With no future supervision, the V-JEPA student is 1.61 points higher than the DINOv3 student (31.45 vs 29.84); with either future target fixed, the V-JEPA student is also consistently higher (35.04 vs 34.43; 38.55 vs 33.79). This indicates that the V-JEPA history representation is more conducive to trajectory generation, but changing the encoder alone accounts for only about 1.6 points.

(ii) The benefit of future supervision depends on whether the future target matches the history representation, and the V-JEPA configuration benefits most. Relative to their respective no-future-supervision baselines, the V-JEPA student paired with the V-JEPA future target yields the largest gain, reaching 38.55, clearly above the other three settings. Our explanation is as follows: V-JEPA features are pretrained on video sequences and encode spatio-temporal representations, so their future-state features carry richer temporal evolution and motion-consistency information. When the history encoder and the future target are of the same origin, the two lie in the same semantic space, allowing the action module to exploit this spatio-temporal consistency constraint more directly for planning and thus obtain the largest gain; when they come from different origins, their semantic spaces are not fully aligned and the additional information is harder to exploit.

(iii) Temporal tokenization is not unified, but its effect points in the opposite direction to the observed advantage. We retain each encoder's original encoding so that each performs at its best: V-JEPA fuses tokens along the temporal dimension and thus has fewer tokens, whereas DINOv3 does no such fusion and thus has more tokens carrying more information. If tokenization or token count dominated performance, the DINOv3 configuration should have benefited; yet in the 2×3 table the V-JEPA student outperforms the DINOv3 student under no future supervision and under both fixed future targets. Thus the context and target confounds are separated by the 2×3 crossing, and although tokenization is not unified, its effect points in the opposite direction to the observed V-JEPA advantage and does not threaten this attribution.

2. Capacity effects in the future-supervision ablation (Weakness 2 / Question 2)

We clarify that in our implementation, the same expert processes both historical visual tokens and future visual tokens, so the model has exactly the same parameter count with and without future supervision. Removing future supervision does not remove an expert or any parameters; it only disables future-state prediction.

3. Efficiency comparison (Weakness 3 / Question 3)

We added an external comparison, selecting DriveWAM, the SOTA baseline in the main table, and measuring it with its public weights under the same hardware and timing conditions:

MethodInference configurationTFLOPs per plan ↓Loaded params (B) ↓Peak memory (GiB) ↓
JEWAM FFaction-only, single forward1.5401.3732.643
DriveWAMvideo 3 steps, action 10 steps17.53512.62724.303

FLOPs are counted with the PyTorch FLOPs counting tool. We currently have no available A800/H100 server and therefore did not measure latency on that hardware; as the table shows, JEWAM is clearly lower than DriveWAM in per-plan computation, loaded parameter count, and peak memory, demonstrating the deployment efficiency advantage of single-forward planning.

4. navtest EC and the single front camera (Weakness 4 / Question 4)

Regarding the navtest EC of 83.50, we performed a scenario-level diagnosis on the NAVSIM benchmark. The EC losses are concentrated in a small number of scenarios: while maintaining progress and smooth driving, the vehicle's acceleration/deceleration and steering adjustments are insufficiently consistent across two consecutive planning executions, reducing extended comfort. After further inspecting the velocity fields and correction processes of these trajectories, we attribute the cause to insufficient cross-window dynamic adjustment in a small subset of trajectories. These failure cases dominate part of the EC sub-metric loss; overall, however, the model retains strong robustness and accuracy and achieves SOTA in EPDMS.

In addition, we further tested on Bench2Drive, where the results (DS 77.95, SR 49.55%) show that our method also has a performance advantage in interactive closed-loop driving:

MethodDS ↑SR ↑
Drive-JEPA64.5236.82
GraphWorld76.7144.22
ReCogDrive71.3645.45
JEWAM77.9549.55

Regarding the single front camera: single-front-view input is a trade-off in our input configuration. We acknowledge that single-front-view input is weak in lateral perception and occlusion handling, and that the paper previously presented it only as an efficiency advantage. It should be noted that our model architecture itself supports multi-view input and can be readily extended to a multi-view setting. Despite the above perceptual limitation, our method using only single-front-view input still outperforms the multi-camera + LiDAR methods in the table:

MethodInput configurationv2 navtest EPDMS ↑v2 navhard EPDMS ↑
TransFuser3 cameras + LiDAR76.7023.10
DiffusionDrive3 cameras + LiDAR84.5027.50
JEWAMsingle front camera90.9839.86

5. Fast-WAM and error bars (Weakness 5 / Question 5)

Fast-WAM is a robotics work, and its direct extension to autonomous driving is Sim-WAM, which explicitly builds on the insight of Fast-WAM. We have compared with Sim-WAM:

MethodParamsNavTest v2 EPDMS ↑NavHard EPDMS ↑
Sim-WAM6B90.237.6
JEWAM FF (ViT-L)0.66B89.5038.55
JEWAM FF (ViT-G)1.3B90.9839.86

With a ViT-L configuration at about 1/10 of Sim-WAM's parameter count, JEWAM already achieves SOTA on NavHard (38.55 vs Sim-WAM 37.6); the ViT-G configuration further improves on both metrics and achieves the best results (NavHard 39.86, NavTest v2 90.98), widening the gap on NavHard relative to ViT-L. We will add this comparison to the paper.

Reviewer_iKe3OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces JEWAM, a joint-embedding world–action model for end-to-end autonomous driving. During training, the model jointly predicts future visual representations and ego trajectories, using an EMA teacher to construct future targets. Asymmetric attention allows the action branch to operate independently at inference. The paper compares five visual representations and selects V-JEPA based on planning performance. It further studies the benefits of future supervision and compares feed-forward and iterative prediction in terms of planning quality and efficiency. Using only front-view camera observations, JEWAM achieves 90.98 and 39.86 EPDMS on NAVSIM v2 NavTest and NavHard, respectively.

Critical strengths
  • Learning from future supervision while directly predicting trajectories at deployment is practically useful. Tables 4 and 5 show a 6.67-point NavHard gain from future supervision, largely retained with action-only inference.
  • The complete system achieves strong benchmark results with efficient inference. Using only front-view observations, JEWAM leads the listed NavTest/NavHard methods and offers the best performance and speed among the evaluated predictor variants.
  • The paper is clearly structured and appropriately qualifies several experimental conclusions. The authors separate the main research questions and acknowledge confounding factors and metric-specific trade-offs.
Critical weaknesses
  • The methodological contribution over the closest prior work needs clearer differentiation. Given the overlap with Metis and Drive-JEPA, matched comparisons would help establish the value beyond framework integration.
  • The representation study does not disentangle stronger historical features from more useful future supervision. Table 3 changes both simultaneously; fixing the context encoder or ablating future supervision across encoders would clarify their contributions.
  • Evaluation remains confined to benchmarks sharing the same underlying data source, without interactive closed-loop validation. Independent closed-loop evaluation would strengthen generalization claims, while lower second-stage progress, lane keeping, and comfort warrant further analysis.
Questions for authors
  • What specifically distinguishes JEWAM from the closest prior work? Please clarify the architectural or training differences from Metis and Drive-JEPA.
  • Does V-JEPA’s advantage depend on joint future-state training? Does its advantage over DINOv3 persist under matched action-only training?
  • Is the action path computationally independent of future-state tokens? Please provide the complete attention mask and explain the small score changes after removing future queries in Table 5.
  • Additional experiment request: Could the authors provide interactive closed-loop evaluations on another dataset or simulation benchmark? This would test whether the reported benefits extend beyond NAVSIM’s pseudo-closed-loop setting.
4 comments

We thank the reviewer for recognizing the training-time future supervision, single-forward deployment, and experimental results, and for the constructive comments. We address each point below.

1. Differences from Metis and Drive-JEPA, and direct comparisons (Weakness 1 / Question 1)

JEWAM differs clearly in design from Drive-JEPA and Metis.

  • Drive-JEPA uses V-JEPA as a visual encoder and performs planning via multi-trajectory distillation and candidate trajectory selection; it does not jointly learn future states and actions and is not a world–action model.
  • Metis builds on the Wan video-generation backbone and jointly learns video generation and action prediction in the latent space of a video VAE, so its training depends on generating/reconstructing pixel-space video. JEWAM adopts and explicitly cites Metis's asymmetric attention design, but instead uses an EMA-updated teacher encoder to encode future observations and construct a self-supervised future-feature target that drives action learning; it does not require the model to generate pixel-space video and uses no pixel reconstruction objective.

The core of JEWAM is thus to jointly learn future states and actions in the encoder feature space: an EMA teacher encodes future observations and constructs a self-supervised future-feature target that directly improves action learning during training, and at deployment a single forward pass outputs the full trajectory. It is essentially a purely latent, self-supervised world–action model. This corresponds to the core claim of the paper: in the WAM task, building an effective history context representation and future-state supervision suffices to obtain planning gains from jointly learning future states and actions, without the model needing video generation capability.

We added direct comparisons with Drive-JEPA and Metis, and Table 1 lists the main-table baseline DriveWAM (the best baseline in the paper) together with the ViT-L and ViT-G versions of JEWAM:

MethodNavTest v2 EPDMS ↑NavHard EPDMS ↑
Drive-JEPA87.80—
Metis90.3032.20
DriveWAM90.5036.10
JEWAM (ViT-L)89.5038.55
JEWAM (ViT-G)90.9839.86

As shown in Table 1, JEWAM (ViT-L) already surpasses all baselines on NavHard (38.55 vs Metis 32.20 and DriveWAM 36.10), i.e., our design achieves SOTA with a smaller parameter count; JEWAM (ViT-G) further improves to 39.86 and achieves SOTA on both evaluation sets. The planning performance of ViT-L and ViT-G is close, differing by only 1.48 and 1.31 points on NavTest v2 and NavHard, respectively, indicating that the final performance does not mainly come from encoder scaling: the contribution of the design itself already holds at the ViT-L scale, and ViT-G brings a further gain. Compared with Metis, JEWAM achieves higher NavHard performance without generating pixel-space video and using only latent self-supervised future features. This direct comparison shows a clear performance advantage of our method over these closely related methods. We will add the above comparisons and the ViT-L results to the main table in the revised manuscript.

2. Separating history representation from future supervision (Weakness 2 / Question 2)

We clarify the controlled variables of the representation study: all compared groups share the same training data, training steps, optimizer, and planning head, and only the combination of history encoder and future-target encoder is changed; the encoder scale is fixed at ViT-L (300M). This setting is already described in the main text.

We conducted a 2×2 cross ablation over the history encoder (student) × future-target encoder (teacher), together with controls in which neither encoder uses future supervision (action-only). Results (NavHard EPDMS) are:

History encoder (student)No future supervisionDINOv3 future targetV-JEPA future target
DINOv3-300M29.8434.4333.79
V-JEPA-300M31.4535.0438.55

The table can be read in two directions:

(i) Encoder quality contributes, but is not the main factor. With no future supervision, the V-JEPA student is 1.61 points higher than the DINOv3 student (31.45 vs 29.84); with either future target fixed, the V-JEPA student is also consistently higher (35.04 vs 34.43; 38.55 vs 33.79). This indicates that the V-JEPA history representation is more conducive to trajectory generation, but changing the encoder alone accounts for only about 1.6 points.

(ii) V-JEPA's advantage persists without future supervision, but its gain is largest under joint future-state training. Under the matched action-only setting (no future supervision), the V-JEPA student already outperforms the DINOv3 student (31.45 vs 29.84); adding future supervision on top, the V-JEPA student paired with the V-JEPA future target yields the largest gain, reaching 38.55, clearly above the other three settings. Our explanation is as follows: V-JEPA features are pretrained on video sequences and encode spatio-temporal representations, so their future-state features carry richer temporal evolution and motion-consistency information. When the history encoder and the future target are of the same origin, the supervisory signal provided by the future features and the history representation lie in the same semantic space, allowing the action module to exploit this spatio-temporal consistency constraint more directly for planning and thus obtain a larger gain. Conversely, when the future target and the history representation come from different pretraining paradigms, their semantic spaces are not fully aligned, and the additional information from future supervision is harder to exploit.

Therefore, the effects of the history encoder and the future target can be read out independently, and the two interact: the spatio-temporal consistency of V-JEPA features makes this match most beneficial. We will add the above 2×3 table to the main text.

3. Computational dependence of the action path and the complete attention mask (Question 3)

We adopt the asymmetric attention design consistent with Metis; the grouped visibility relations are as follows (rows = Query, columns = Key/Value, 1 = allowed, 0 = masked):

Query / Key–ValueHistory and conditioning tokensFuture-state tokensAction tokens
History and conditioning tokens100
Future-state tokens111
Action tokens101

Action tokens and history/conditioning tokens cannot read future-state tokens, so future states can neither directly affect actions nor indirectly affect actions through the history context; future-state queries can read action tokens, so future-feature prediction is conditioned on actions during training. The action path is therefore computationally independent of future-state tokens. After removing future queries, the small score changes in Table 5 arise from numerical differences in the forward computation, while overall planning performance remains stable, consistent with the conclusion that deployment does not require explicit future-state estimation. We will add the complete attention mask figure to the revised manuscript.

4. Independent closed-loop evaluation and NavHard sub-metric analysis (Weakness 3 / Question 4)

To address the concern that evaluation is confined to benchmarks sharing the same underlying data source and lacks interactive closed-loop validation, we added an independent closed-loop evaluation on Bench2Drive. We emphasize that Bench2Drive and NAVSIM do not share the same underlying data source, and that Bench2Drive provides an interactive closed-loop simulation environment with dynamic traffic participants and continuous decision-making, thus testing generalization across data sources and under interactive closed-loop conditions. Setup: starting from the JEWAM FF model with a V-JEPA-G encoder trained on NAVSIM, we fine-tuned on the Bench2Drive Base training set for 3,000 steps with a global batch size of 128, keeping the input information consistent with NAVSIM training; the model uses a single front camera and outputs the full trajectory in a single forward pass. Results on the official test set are:

MethodDS ↑SR ↑
Drive-JEPA64.5236.82
GraphWorld76.7144.22
ReCogDrive71.3645.45
JEWAM77.9549.55

With single-forward planning, JEWAM achieves the highest DS and SR among the compared methods, showing a performance advantage in interactive closed-loop driving on a non-overlapping data source as well. We will add this experiment in the final version.

Regarding the low EP, LK, and EC on NavHard stage 2 pointed out by the reviewer, we conducted a systematic analysis of the NavHard benchmark and found that the model exhibits anomalies in the EP, LK, and EC metrics in some scenarios. Further trajectory visualization and diagnosis show that some vehicles, while maintaining progress and smooth driving, fail to effectively correct existing lateral deviations, leading to lane-keeping failures. By analyzing the velocity fields and trajectory corrections of these trajectories, we found insufficient speed recovery and insufficient bias correction in a small subset of trajectories. These failure cases dominate part of our sub-metric performance on NavHard; nevertheless, our model still demonstrates strong robustness and accuracy, achieving SOTA in EPDMS.

Reviewer_iKe3OpenReview ↗

Maintaining Weak Accept

Thank you for the detailed rebuttal. The cross-encoder ablations and the explanation of the full attention mask address my main questions about the method. The additional closed-loop results on Bench2Drive also make a stronger case for using future supervision during training while keeping deployment efficient. I still see the methodological novelty as limited, and the driving metrics show some trade-offs. On balance, I’m keeping my Weak Accept rating. Please include the new results and the full experimental settings in the revised paper.

Reviewer_YPUgOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces JEWAM, a driving planner trained to predict future visual representations and ego trajectories jointly. During training, an EMA teacher encodes future observations as supervision. At inference, the future-prediction branch is removed, and the model predicts the full trajectory in one forward pass.

Critical strengths
  1. JEWAM achieves the highest score among the methods listed (90.98 EPDMS on NAVSIM v2 NavTest and 39.86 on NavHard). It can predict future trajectory in one forward pass without the need to generate future states at inference.
  2. The paper uses future observations as an additional supervision signal in training, and the ablation showed that removing the future-state branch has little effect on planning score, supporting the practicality of this lightweight design.
Critical weaknesses
  1. The proposed method seems to combine a few well-known ideas in prior work. Future prediction in latent space has been explored by Drive-JEPA and Latent-WAM. The asymmetric attention design has been explored by Metis. Can the author show what is new in JEWAM, and give a comparison with these methods on the performance?
  2. The representation comparison changes a few factors at the same time, making it difficult to see why V-JEPA performs better. For example, the pretraining, temporal tokenization, historical features, and future prediction targets. Also, the larger ViT-G encoder differs from the ViT-L used in the representation study. A cleaner comparison can help to clarify if the better performance comes from the design or other factors.
Questions for authors
  1. How does JEWAM differ from Drive-JEPA and Metis in its design and training? Can the author provide a direct comparison with these methods on NAVSIM?
  2. Can the author provide a clearer representation study that shows the effect of the historical encoder being separated from that of the future prediction target? For example, what happens when the historical encoder is fixed and only the target representation changes?
  3. How much does the larger ViT-G encoder contribute to the final performance? How does the ViT-L version of JEWAM compare with the baselines in the main results tables?
3 comments

We thank the reviewer for the careful review and constructive comments. These suggestions help us state our contributions more clearly and improve the experimental comparisons. We address each point below.

1. Differences from Drive-JEPA and Metis, and direct comparisons (Weakness 1 / Question 1)

JEWAM differs clearly in design from Drive-JEPA and Metis.

  • Drive-JEPA uses V-JEPA driving-video pretraining to obtain visual representations, and its planning core is multi-trajectory distillation and candidate trajectory selection; it does not jointly learn future states and actions.
  • Metis relies on the Wan backbone and jointly learns video generation and action prediction in the latent space of a video VAE, so its training depends on generating/reconstructing pixel-space video. JEWAM adopts and explicitly cites Metis's asymmetric attention design, but instead uses an EMA-updated teacher encoder to encode future observations and construct a self-supervised future-feature target that drives action learning; it does not require the model to generate pixel-space video and uses no pixel reconstruction objective.

The core of JEWAM is thus to jointly learn future states and actions in the encoder feature space: an EMA-updated teacher encoder provides a self-supervised future-feature target, encouraging the model to learn future dynamics relevant to action decisions, and at inference a single forward pass outputs the trajectory. It is essentially a purely latent, self-supervised world–action model. This corresponds to the core claim of the paper: in the WAM task, building an effective history context representation and future-state supervision suffices to obtain planning gains from jointly learning future states and actions, without the model needing video generation capability.

We added direct comparisons with Drive-JEPA and Metis, and Table 1 lists the main-table baseline DriveWAM (the best baseline in the paper) together with the ViT-L and ViT-G versions of JEWAM:

MethodNavTest v2 EPDMS ↑NavHard EPDMS ↑
Drive-JEPA87.80—
Metis90.3032.20
DriveWAM90.5036.10
JEWAM (ViT-L)89.5038.55
JEWAM (ViT-G)90.9839.86

As shown in Table 1, JEWAM (ViT-L) already surpasses all baselines on NavHard (38.55 vs Metis 32.20 and DriveWAM 36.10), i.e., our design achieves SOTA with a smaller parameter count; JEWAM (ViT-G) further improves to 39.86 and achieves SOTA on both evaluation sets. The planning performance of ViT-L and ViT-G is close, differing by only 1.48 and 1.31 points on NavTest v2 and NavHard, respectively, indicating that the final performance does not mainly come from encoder scaling: the contribution of the design itself already holds at the ViT-L scale, and ViT-G brings a further gain. Compared with Metis, JEWAM achieves higher NavHard performance without generating pixel-space video and using only latent self-supervised future features. This direct comparison shows a clear performance advantage of our method over these closely related methods. We will add the above comparisons and the ViT-L results to the main table in the revised manuscript.

2. Confounding factors and separability in the representation study (Weakness 2 / Question 2)

We clarify the controlled variables of the representation study: all compared groups share the same training data, training steps, optimizer, and planning head, and only the combination of history encoder and future-target encoder is changed; the encoder scale is fixed at ViT-L (300M). The study therefore addresses "how the history representation and the future target match under a fixed scale and training configuration," rather than a cross-scale comparison. This setting is already described in the main text.

We conducted a 2×2 cross ablation over the history encoder (student) × future-target encoder (teacher), together with controls in which neither encoder uses future supervision. Results (NavHard EPDMS) are:

History encoder (student)No future supervisionDINOv3 future targetV-JEPA future target
DINOv3-300M29.8434.4333.79
V-JEPA-300M31.4535.0438.55

The table can be read in two directions:

(i) Encoder quality contributes, but is not the main factor. With no future supervision, the V-JEPA student is 1.61 points higher than the DINOv3 student (31.45 vs 29.84); with either future target fixed, the V-JEPA student is also consistently higher (35.04 vs 34.43; 38.55 vs 33.79). This indicates that the V-JEPA history representation is more conducive to trajectory generation, but changing the encoder alone accounts for only about 1.6 points.

(ii) The benefit of future supervision depends on whether the future target matches the history representation, and the V-JEPA configuration benefits most. Relative to their respective no-future-supervision baselines, the V-JEPA student paired with the V-JEPA future target yields the largest gain, reaching 38.55, clearly above the other three settings. Our explanation is as follows: V-JEPA features are pretrained on video sequences and encode spatio-temporal representations, so their future-state features carry richer temporal evolution and motion-consistency information. When the history encoder and the future target are of the same origin, the supervisory signal provided by the future features and the history representation lie in the same semantic space, allowing the action module to exploit this spatio-temporal consistency constraint more directly for planning and thus obtain a larger gain. Conversely, when the future target and the history representation come from different pretraining paradigms, their semantic spaces are not fully aligned, and the additional information from future supervision is harder to exploit.

Therefore, the effects of the history encoder and the future target can be read out independently, and the two interact: the spatio-temporal consistency of V-JEPA features makes this match most beneficial. We will add the above 2×3 table to the main text.

These results are consistent with the core claim of the paper: WAM can improve action learning through effective history representations and future-feature supervision without relying on video generation.

Thank you for the additional comparisons and cross-ablation. The new results substantially address my concerns about separating the history encoder from the future-target representation. I am updating my recommendation to weak acceptance.

Reviewer_Ms3HOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces JEWAM, a joint-embedding world–action model that jointly learns future visual representations and driving trajectories. Given four historical front-view images and the current ego state, the model predicts eight future BEV poses over a four-second horizon. During training, an EMA teacher provides future representation targets, and an asymmetric attention mechanism allows future-state queries to attend to action tokens while preventing the reverse dependency. The teacher and future-state branch can therefore be removed at deployment, allowing full-horizon trajectory prediction in a single forward pass.

Critical strengths
  1. Learning from future observations while retaining a simple deployment path addresses an important trade-off between planning quality and inference cost. The distinction between training-time supervision and inference-time computation is clearly motivated and implemented.

  2. The manuscript distinguishes the ViT-L representation study from the ViT-G final model and explicitly acknowledges that the encoder comparison evaluates complete configurations and that the iterative variants also change the training objective. These qualifications make the empirical findings easier to interpret.

Critical weaknesses
  1. The contribution is primarily empirical, and the representation analysis could provide deeper insight. Based on the manuscript’s own positioning, several central ingredients, including asymmetric routing and training-time future prediction with action-only deployment, build on existing approaches. The contribution therefore rests substantially on the empirical findings. However, changing the encoder simultaneously changes historical context features and future targets, alongside pretraining and tokenization differences. The results identify a strong configuration but do not establish whether V-JEPA is especially beneficial for future-supervised action learning or primarily provides a stronger observation encoder. A comparison of at least DINOv3 and V-JEPA both with and without future supervision would make the contribution more informative.

  2. The efficiency results need more complete measurement details. The reported 29.10 Hz and speedup over the internal iterative variants are useful, but batch size, numerical precision, timing boundaries, parameter count, and peak memory usage are not specified sufficiently clearly. These details are important for assessing deployment cost, particularly given the ViT-G backbone. A comparison with an external baseline under matched hardware and timing conditions would further contextualize the efficiency claim.

  3. Despite the higher aggregate NavHard score, JEWAM has lower second-stage progress, lane keeping, and extended comfort than the compared DriveWAM baseline; extended comfort is 60.50 versus 72.80. Scenario-level breakdowns and representative failure cases would help explain these trade-offs. More broadly, the evidence is confined to NAVSIM, and the two-stage pseudo-closed-loop protocol does not establish long-horizon interactive driving robustness.

Questions for authors
  1. Could you compare DINOv3 and V-JEPA both with and without future supervision to determine whether the representation choice changes the benefit of the auxiliary task, rather than only the quality of observation encoding?

  2. What explains the same-checkpoint Future On/Off differences in Tables 3 and 5? Can historical context tokens be updated using future-state tokens? Providing the complete attention mask and trajectory-level differences would help distinguish numerical effects from additional computational dependencies.

  3. What behaviors account for the lower second-stage comfort, progress, and lane-keeping scores? Could you provide scenario-level analysis or representative failure cases?

3 comments

We thank the reviewer for their careful review of this paper and for acknowledging the design of training-time future supervision and single-forward-pass deployment. Below we respond to each comment point by point.

1. How the choice of representation affects the benefit of future supervision (Weakness 1 / Question 1)

We thank the reviewer for pointing out the need to distinguish between "a stronger historical observation encoder" and "more effective future supervision." As suggested, we conducted an additional 2×2 cross experiment between the history encoder (student) and the future target encoder (teacher), with a unified encoder scale (300M), and added experiments with these two encoders under no future supervision. The final results are as follows (NavHard EPDMS):

History encoder (student)No future supervisionDINOv3 future targetV-JEPA future target
DINOv3-300M29.8434.4333.79
V-JEPA-300M31.4535.0438.55

This table can be read from two directions:

(i) Encoder quality itself contributes, but is not the main factor. Holding "no future supervision" fixed, the V-JEPA student is 1.61 points higher than the DINOv3 student (31.45 vs. 29.84); holding either future target fixed, the V-JEPA student is also consistently higher (35.04 vs. 34.43; 38.55 vs. 33.79). This shows that V-JEPA history representations are more conducive to trajectory generation, but changing the encoder alone can only account for about 1.6 points.

(ii) The benefit of future supervision depends on whether the future target matches the history representation, and the V-JEPA configuration benefits the most. Taking each configuration's no-future-supervision result as its baseline, the V-JEPA student paired with the V-JEPA future target yields the largest gain, reaching 38.55, clearly higher than the other three groups. Our explanation is as follows: V-JEPA features learn spatiotemporal representations of video sequences during pretraining, and their future-state features carry richer information about temporal evolution and motion consistency; when the history encoder and the future target come from the same source, the supervision signal provided by the future features and the history representation lie in the same semantic space, allowing the action module to more directly exploit this spatiotemporal consistency constraint to learn planning, and thus yielding a larger gain. Conversely, when the future target and the history representation come from different pretraining paradigms, their semantic spaces are not fully aligned, and the additional information brought by future supervision is harder to exploit fully.

Therefore, V-JEPA does not merely provide a stronger observation encoder; the benefit of future supervision depends on the match between future features and history representations, and V-JEPA's spatiotemporal consistency features make the gain from this match the most pronounced. This is consistent with the core argument of this paper—WAM can improve action learning through effective history representations and future feature supervision, without relying on video generation capability. We will add the above 2×3 table to the main text.

2. Inference efficiency, parameter count, and peak memory (Weakness 2)

We thank the reviewer for pointing out that the efficiency measurement details are insufficient. The measurement configuration is: batch size 1, BF16 precision, a single Alibaba Zhenwu PPU-ZW810E. JEWAM's speed only counts one complete forward pass of the model: after 5 warm-up runs, the average forward time measured on NavTest is about 34.36 ms, corresponding to 29.10 Hz; the timing boundary is the model forward pass, excluding data loading and post-processing. The parameter count is the number of loaded parameters, and peak memory is the peak value of a single forward pass.

Under the same hardware and timing conditions, we selected DriveWAM, the best-performing open-source baseline in the main table, and used its public weights for an external comparison:

MethodInference configurationFrequency (Hz)↑TFLOPs per planning↓Loaded parameters (B)↓Peak memory (GiB)↓
JEWAM FFaction-only, single forward pass29.101.5401.3732.643
DriveWAMvideo 3 steps, action 10 steps3.8917.53512.62724.303

The computation is counted with the PyTorch FLOPs counting tool. As shown in the table, under the same hardware and timing conditions, JEWAM clearly outperforms DriveWAM in inference frequency, computation per planning, loaded parameter count, and peak memory, demonstrating the deployment efficiency advantage of single-forward-pass planning.

3. NavHard sub-metric trade-off, failure cases, and closed-loop validation (Weakness 3 / Question 3)

We conducted a systematic analysis of the NavHard benchmark and found that the model's EP, LK, and EC metrics are anomalous in some scenarios. Further trajectory visualization and diagnosis showed that some vehicles, while maintaining progress and smooth driving, failed to effectively correct existing lateral deviations, leading to lane-keeping failures. By analyzing the velocity fields and trajectory correction of these trajectories, we found insufficient speed recovery and insufficient bias correction in a small subset of trajectories. These failure cases dominate our performance on some sub-metrics in NavHard, but despite this, our model still demonstrates strong robustness and accuracy, achieving state-of-the-art EPDMS results.

To further examine the method's performance in continuous interaction, we added closed-loop experiments on Bench2Drive: starting from JEWAM FF with the V-JEPA-G encoder trained on NAVSIM, we fine-tuned for 3,000 steps on the Bench2Drive training set with a global batch size of 128; the model uses a single front-view camera and outputs the complete trajectory in a single forward pass. Official test set results:

MethodDS ↑SR ↑
Drive-JEPA64.5236.82
GraphWorld76.7144.22
ReCogDrive71.3645.45
JEWAM77.9549.55

JEWAM achieves the highest DS and SR among the compared world-model-based methods with single-forward-pass planning, and likewise demonstrates a performance advantage in closed-loop interactive driving. We will add a Bench2Drive experiment subsection to the main text; these results show that the proposed method also has a performance advantage in closed-loop interactive driving.

4. Future On/Off difference under the same checkpoint and attention mask (Question 2)

We follow Metis's asymmetric attention design, with the group visibility relations as follows (rows = Query, columns = Key/Value, 1 = allowed, 0 = masked):

Query / Key–ValueHistory and condition tokensFuture state tokensAction tokens
History and condition tokens100
Future state tokens111
Action tokens101

That is, future-state queries can read action tokens, action queries do not read future-state tokens, and history and condition tokens also do not read future-state tokens; therefore, there is no attention pathway through which future states could influence action prediction via the history context. Under the same checkpoint, the small difference between Future On/Off comes from numerical differences in the forward computation.

Thanks for the response. These answers have addressed my concerns.

Comments on the paper

1 comment
Area_Chair_DyenOpenReview ↗

Reminder: Author Response and Reviewer Discussion

Dear Reviewers,

We are halfway through the discussion period. Please review the authors’ responses, address any remaining concerns, and engage in the discussion where appropriate. If your assessment remains unchanged, please briefly acknowledge that you have considered the authors’ responses.

Thank you for your time and contributions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.