acceptodds
Under review as a conference paper at ICLR 2027

DA-WAM: Decision-Aligned Future Latents for Driving World Models

16 views

Abstract

World models for autonomous driving should anticipate the consequences that matter for driving decisions rather than reconstruct every detail of future scenes. However, predictive representations alone are not guaranteed to distinguish candidate actions that lead to different planning outcomes. We propose DA-WAM, a decision-aligned world action model that couples a Joint-Embedding Predictive Architecture (JEPA) with candidate-level trajectory scoring. During joint training, the JEPA objective grounds future representations in observed scene dynamics, while the planning objectives encourage these representations to capture distinctions relevant to safety and utility. An action-conditioned predictor generates a distinct future latent for each candidate trajectory, and a multi-head trajectory scorer uses the corresponding latent to estimate planning factors and overall utility. In addition, safety-critical post-training with hard negatives further improves discrimination between geometrically similar trajectories with different safety outcomes. DA-WAM achieves state-of-the-art performance among learned camera-based planners on NAVSIM-v1, NAVSIM-v2, and Bench2Drive. Ablation studies demonstrate the effectiveness of each component in driving decision-making.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
autonomous driving, world models, trajectory planning, action-conditioned prediction, joint-embedding predictive architecture, representation learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_iQyGOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DA-WAM, a decision-aligned world action model for autonomous driving. It predicts a distinct compact future representation for each candidate trajectory and combines it with the current scene context to assess trajectory safety and driving utility, ultimately selecting a driving trajectory. Its main innovation is a compact candidate representation that supports both future prediction and trajectory scoring. Joint optimization with dense future-feature supervision and planning objectives during training encourages the representations to capture scene dynamics relevant to decision-making. The model further incorporates hard-negative trajectories that are geometrically similar but have different safety outcomes to sharpen safety boundaries. Evaluated on NAVSIM-v1, NAVSIM-v2, and Bench2Drive, DA-WAM achieves 93.7 PDMS, 89.2 EPDMS (with temporal-consistency reranking), and a Driving Score of 59.04, respectively.

Critical strengths
  1. The paper presents a concrete and efficient way to connect future prediction with trajectory scoring. DA-WAM scores each candidate trajectory using its own compact representation and applies future-feature supervision to the expert-matched candidate during joint training. At inference, the dense decoder is removed while the candidate-conditioned prediction branch remains active. This is a reasonable design that integrates predictive supervision into planning without reconstructing a complete future scene for every candidate, offering practical value.
  2. The experimental evaluation is relatively comprehensive, covering NAVSIM-v1, NAVSIM-v2, and Bench2Drive to assess both planning metrics and closed-loop driving performance. The targeted safety refinement also yields measurable improvements.
Critical weaknesses
  1. The contribution beyond WoTE is insufficiently established. WoTE already predicts candidate-specific future BEV states for reward evaluation, so the claimed limitations of shared futures or weak candidate–future coupling need clarification. Controlled comparisons are needed to isolate the benefits of DA-WAM’s representation, supervision, and safety refinement.
  2. The role of future representations in decision-making is insufficiently validated. The experiments do not clearly distinguish whether the gains arise from modeling future consequences or from stronger scene–trajectory feature interactions. Equal-capacity controls and trajectory–future representation shuffling experiments would help validate the proposed mechanism.
Questions for authors
  1. WoTE already predicts candidate-specific future states for trajectory evaluation. What specific limitation in WoTE’s candidate–future coupling does DA-WAM address, and how does its architecture overcome it?
  2. Do the “No Future Prediction” and “Action-Conditioned Future” variants in Table 4 have comparable model capacity, scoring architectures, and training settings? What existing evidence distinguishes gains from future-prediction supervision from those due to additional scene–trajectory feature processing?
2 comments
AuthorsOpenReview ↗

Response to Reviewer iQyG

We thank the reviewer for identifying two questions that helped us clarify our contribution: how DA-WAM differs from WoTE, and what evidence shows that future prediction contributes to trajectory selection. Our responses to each point are provided below.

W1 / Q1: Contribution beyond WoTE

We acknowledge that WoTE already performs candidate-specific future BEV prediction and reward evaluation.

Our contribution concerns how each candidate’s future representation is learned. WoTE supervises future BEV states with predefined semantic maps. Similar to image or occupancy reconstruction, the target specifies the scene details that the model should retain, including those that have a relatively small impact on the decision. In DA-WAM, observed future features supervise the representation of the expert-matched candidate, while candidate-level planning objectives train the model to evaluate all candidates. During training, gradients from trajectory evaluation also update the future predictor. This is what makes the representation decision-aligned: the planning objectives help determine which distinctions about safety and progress the representation retains.

We have improved Figure 1 to make this distinction explicit. Figure 1(a) illustrates predefined prediction targets, including the semantic maps used by WoTE, and Figure 1(c) shows how predictive and planning supervision jointly shape DA-WAM’s representations. The corresponding revisions in the Introduction and Related Work are highlighted in blue.

In order to better highlight our advantages, we conducted a comparison on navsim:

MethodInputsNC ↑DAC ↑TTC ↑Comfort ↑EP ↑PDMS ↑
WoTECameras + LiDAR98.5096.8094.9099.9081.9088.30
DA-WAM (five-seed mean)Cameras99.1298.9096.8199.7689.9993.70

W2 / Q2: Equal-Capacity Control and Future-Supervision Effects

(1) Equal-capacity comparison under identical Stage 2 refinement.
We apologize for the ambiguity in our original description. The statement around Line 394, “Δ=0\Delta=0 predicts the current latent state,” refers to our architecture-matched control. Specifically, Current-Latent Conditioning (Δ=0\Delta=0) and Action-Conditioned Future (Δ=0.5\Delta=0.5) share the same architecture and parameter capacity, including the predictor, candidate-conditioned feature processing, and scorer. Only the prediction target differs: the former predicts the current latent representation, whereas the latter predicts the future latent representation. Current-Latent Conditioning therefore serves as the equal-capacity baseline for Action-Conditioned Future. In contrast, the No-Future variant removes the prediction module and consequently has a different architecture.

To distinguish the benefit of future supervision from that of safety refinement, we evaluate both variants before and after the same Stage 2 procedure:

ConfigurationWithout Stage 2With Stage 2Stage 2 Gain
Current-Latent Conditioning (Δ=0\Delta=0)93.2593.51+0.26
Action-Conditioned Future (Δ=0.5\Delta=0.5)93.4693.68+0.22
Gain+0.21+0.17—

Future supervision improves PDMS by 0.21 before Stage 2 and 0.17 after Stage 2, demonstrating a benefit under both settings with model capacity held constant. Stage 2 also improves both variants, by 0.26 for Current-Latent Conditioning and 0.22 for Action-Conditioned Future. These comparisons distinguish the contribution of the future prediction target from the additional benefit of safety refinement.

(2) Trajectory–latent alignment test.
To assess whether the trained scorer relies on the correspondence between each candidate trajectory and its predicted latent, we perform the requested shuffling test without retraining:

Pairing ConditionPDMS ↑Δ PDMS
Original pairing93.6806—
Joint permutation of trajectories and latents93.68060.0000
Latent swapping between similar trajectories91.0207−2.6599
Random latent swapping, five-trial mean90.5967−3.0839
Latent swapping between dissimilar trajectories88.5346−5.1460

Jointly permuting trajectories and latents preserves their correspondence and leaves PDMS unchanged. In contrast, breaking this correspondence reduces PDMS by 2.66–5.15 points, with the largest decrease occurring when latents are swapped between dissimilar trajectories. This pattern indicates that the scorer uses candidate-specific trajectory–latent alignment rather than treating predicted latents as interchangeable context. The equal-capacity comparison in (1) further shows the benefit of using future targets to learn these representations.

AuthorsOpenReview ↗

Additional Response to Reviewer iQyG (Text Revision Instructions)

Dear Reviewer,

We found your comments especially insightful. They helped us substantially improve the manuscript’s readability and explain our contribution more clearly. We made the following revisions.

Q1. We revised Figure 1 and clarified in the Introduction and Related Work sections how DA-WAM and WoTE differ in the way they link candidate trajectories to predicted futures.

Q2. We clarified the comparison of model capacity at line 377 and reported the results of the trajectory–latent alignment test in Table 9. Figure 6 and Appendix Figures 12–13 show how different candidate actions yield different predicted futures from the same observation.

We would be very pleased if these revisions address your concerns. Please feel free to let us know if any questions or concerns remain. We would be happy to provide further clarification.

Best regards,

The Authors of Submission 16157

Reviewer_meU9OpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents DA-WAM, a decision-aligned world action model for autonomous driving that integrates future latent prediction with candidate-level trajectory evaluation. The method builds on a pretrained V-JEPA 2.1 visual encoder and employs an action-conditioned predictor to generate a separate compact latent representation for each candidate trajectory. These representations are combined with trajectory and scene features to estimate planning factors and overall utility. During training, a dense future-feature prediction objective supervises the candidate closest to the expert trajectory, while planning losses provide supervision for other candidates. The framework further introduces safety-focused post-training using hard-negative trajectories to improve candidate discrimination. Experiments on NAVSIM-v1, NAVSIM-v2, and Bench2Drive report competitive performance. Although the overall framework is reasonably well motivated, I have concerns about whether the learned representations genuinely capture action-dependent future consequences and whether the experimental evidence supports the claimed advantages.

Critical strengths
  1. The paper addresses a meaningful limitation of existing driving world models by emphasizing the relationship between candidate-specific future representations and trajectory evaluation.

  2. The combination of predictive representation learning, candidate-conditioned scoring, and safety-focused supervision forms a coherent framework with a reasonably clear training and inference pipeline.

  3. The evaluation includes several established autonomous driving benchmarks, along with architectural ablations, parameter sensitivity studies, and qualitative representation analyses.

Critical weaknesses
  1. The paper does not convincingly demonstrate that the candidate-conditioned representations actually model different future outcomes. The central contribution is the one-to-one association between each candidate trajectory and its predicted future latent. However, according to Section 3.1, these representations are produced by applying trajectory-conditioned attention to current scene features. While this ensures that different candidates can obtain different feature embeddings, it does not establish that these embeddings capture the physical consequences of executing different actions. More importantly, direct future-feature supervision is available only for the expert-matched candidate, while other candidates receive planning-derived labels without observed counterfactual future targets. The resulting representations may therefore function primarily as trajectory-conditioned scoring features rather than genuine predictive future states. The qualitative analyses do not resolve this concern, since they visualize the dense reconstruction pathway rather than establishing the counterfactual predictive content of each compact candidate representation.

  2. The expert-matching strategy can introduce incorrect supervision into the future-prediction objective. Section 3.2 assigns the observed future representation to the generated candidate with the smallest average displacement error relative to the expert trajectory. However, the closest candidate is not necessarily sufficiently close to the executed expert action, especially in challenging scenes where the proposal generator performs poorly. The implementation explicitly applies this supervision without a maximum matching-error threshold. As a result, the future-prediction objective can associate an observed future with a trajectory that would have produced a meaningfully different outcome.

  3. The ablation results provide rather limited evidence for the key architectural contribution. In Table 4, action-conditioned future prediction achieves 93.46 PDMS before safety refinement, compared with 93.31 for the no-future baseline. This improvement of only 0.15 points is relatively small, especially considering the additional predictor, dense decoder, and predictive supervision introduced by the method. The final safety-focused refinement increases performance from 93.46 to 93.68, but this stage simultaneously modifies external candidate sampling, unsafe-candidate weighting, and pairwise ranking supervision. Their individual contributions are not adequately separated. Moreover, the no-future and shared-future alternatives are not evaluated under the same final safety-refinement setting in Table 4.

  4. The closed-loop results reveal important trade-offs that are not sufficiently analyzed. Table 3 reports the highest Bench2Drive Driving Score of 59.04, but the advantage over SeerDrive is only 0.72 points. More importantly, DA-WAM achieves a Success Rate of 24.66%, below SeerDrive's 30.17%, and its Comfortness score of 14.64 is substantially lower than those of several reported baselines. Its average trajectory displacement error is also worse than that of several competitors. These results suggest that the improvement in the aggregate Driving Score does not necessarily translate into more reliable or comfortable closed-loop driving.

Questions for authors
  1. What evidence demonstrates that the compact candidate-conditioned latents encode future scene evolution rather than simply providing additional trajectory-conditioned features to the scorer? Could you evaluate counterfactual prediction fidelity or perform a candidate-latent swapping experiment?

  2. How large is the ADE between the expert trajectory and the closest generated candidate during training? Have the authors investigated whether excluding poorly matched candidates or weighting predictive supervision by matching quality improves performance?

  3. Could the authors report the final performance of the no-future and shared-future baselines under the same Stage 2 refinement procedure? It would also be helpful to isolate the effects of hard-negative sampling, unsafe-candidate weighting, and pairwise ranking.

  4. What is the NAVSIM-v2 performance without temporal-consistency reranking? Additionally, could the authors explain the relatively low Bench2Drive Success Rate and Comfortness scores despite achieving the highest aggregate Driving Score?

5 comments
AuthorsOpenReview ↗

Response to Reviewer meU9 1/3

We thank the reviewer for the careful analysis of predictive grounding, expert matching, causal attribution, and closed-loop performance. We provide additional measurements and clarify the scope of the claims supported by them.

W1 / Q1: Predictive information versus trajectory-conditioned features

We agree that trajectory-conditioned attention alone does not establish predictive semantics. In our method, the future-feature objective grounds the expert-matched representation in an observed future target. Other candidates receive planning-derived supervision, not observed counterfactual future targets.

The design separates predictive grounding from decision alignment: observed future features supervise the expert-matched candidate, while explicit planning-factor and utility targets supervise candidate evaluation. This gives the compact representations a predictive target and the scorer a decision-quality target, without assuming observed futures for every candidate.

We therefore distinguish two questions: (1) whether the predictor improves over copying the current state and (2) whether the scorer uses the candidate–latent association.

(1) For the first question, we obtain:

Prediction horizonPredicted L1 ↓Copy-current L1 ↓Relative reduction ↑
0.5 s0.39550.495920.25%
1.0 s0.45100.626127.97%
1.5 s0.49480.708430.15%

These measurements show improvement over copying current features on available future targets. They are not measurements of unobserved counterfactual outcomes for all candidates. The architecture-matched Current-Latent comparison below separately tests the contribution of the prediction target.

(2) For the second question, we performed the requested swapping experiment while keeping the trained model fixed:

Pairing conditionPDMS ↑Δ PDMSFuture latent L1 ↓
Original pairing93.6806—0.336550
Joint permutation of trajectories and latents93.68060.00000.336550
Latent swapping between similar trajectories91.0207−2.65990.337297
Random latent swapping, five-trial mean90.5967−3.08390.342251
Latent swapping between dissimilar trajectories88.5346−5.14600.346909

Joint permutation leaves PDMS unchanged, whereas mismatching latents reduces PDMS by 2.66–5.15 points, with the largest drop for dissimilar trajectories. Thus, candidate-associated representations materially influence the trained scoring pathway.

W2 / Q2: Expert matching and potentially incorrect supervision

We agree that selecting the closest proposal does not guarantee an exact match to the executed expert trajectory. We measured the mean ADE of the closest candidate throughout training:

Epoch0149151929
Mean matching ADE ↓1.30760.47960.35970.31080.27650.26500.2441

Mean matching error decreases by approximately 81.3%, indicating progressively better average alignment between the supervised candidate and the expert action. However, neither a low mean ADE nor a small geometric deviation guarantees equivalent physical outcomes near a collision boundary.

The swapping results provide a complementary observation: preserving candidate–latent correspondence is useful for decisions under the current training procedure.

AuthorsOpenReview ↗

Response to Reviewer meU9 2/3

W3 / Q3: Equal-capacity controls and Stage 2 attribution

(1) We apologize for the ambiguity in our original description. Current-Latent Conditioning (Δ=0\Delta=0) and Action-Conditioned Future (Δ=0.5\Delta=0.5) share the same architecture and parameter capacity, including the predictor, candidate-conditioned feature processing, and scorer. The only difference is the prediction target: the former predicts the current latent representation, whereas the latter predicts the future latent representation. Thus, Current-Latent Conditioning is the equal-capacity baseline for Action-Conditioned Future. The No-Future variant removes the prediction module and therefore does not share the same architecture.

Both variants are evaluated with the same Stage 2 refinement:

ConfigurationWithout Stage 2With Stage 2Stage 2 gain
Current-Latent Conditioning (Δ=0\Delta=0)93.2593.51+0.26
Action-Conditioned Future (Δ=0.5\Delta=0.5)93.4693.68+0.22
Gain+0.21+0.17—

The +0.17 PDMS difference after identical refinement addresses whether the future target offers value beyond capacity and safety training.

(2) We also isolate the three Stage 2 components:

MethodPDMS ↑Δ PDMSNOC ↑DAC ↑Progress ↑TTC ↑Comfort ↑
Initial93.46—98.8898.5890.4796.3399.69
Ranking Loss Only92.62−0.8498.0198.0791.8093.9199.55
Hard-Negative Sampling Only93.65+0.1999.1598.8689.8396.8899.79
Unsafe-Candidate Weighting Only93.59+0.1399.1498.8589.7296.8799.82
All (Full Stage 2)93.70+0.2499.1298.9089.9996.8199.76

Ranking loss alone increases Progress by 1.33 points, but degrades safety metrics and reduces PDMS by 0.84. This shows a changed safety–progress balance, not that a lower training loss necessarily implies better planning. Hard-negative sampling and unsafe weighting independently improve PDMS by 0.19 and 0.13, respectively, alongside improved safety metrics and lower Progress. The full configuration has the highest listed PDMS.

(3) The manuscript's attention visualizations additionally show stronger responses around interacting vehicles and traffic signals after safety refinement. This is qualitative evidence consistent with more decision-relevant feature use, whereas the component ablations and five-seed measurements quantify the observed planning gains.

For the complete refinement, the five-seed comparison is:

ConfigurationPDMS, mean ± std ↑
Action-Conditioned Future without safety93.4733 ± 0.0225
Action-Conditioned Future with safety93.7039 ± 0.0072
Mean improvement+0.2306

The mean-to-mean gain is 0.2306; the approximately +0.24 in the component table instead uses the original Initial checkpoint as its reference.

Manuscript revision. We identify Δ=0\Delta=0 as the equal-capacity control, report Stage 2 components separately, and state explicitly that Stage 2 adds safety-oriented supervision while turning off the future-feature loss. We attribute future-target gains to the controlled comparison and refinement gains to the refinement experiments.

W4 / Q4: Bench2Drive and the scoring-objective gap

We acknowledge the trade-offs. DA-WAM achieves 59.04 Driving Score, versus 58.32 for SeerDrive, but its 24.66% Success Rate is below SeerDrive's 30.17%, and its 14.64 Comfortness indicates a limitation. Our trajectory displacement error is also not the best in the comparison. However, results demonstrate a substantial efficiency advantage alongside a relatively modest driving-score gap, supporting the competitiveness of our approach on Bench2Drive.

A higher aggregate score does not require a higher fraction of completely successful routes: route-level partial progress and penalties can affect the aggregate differently from an all-or-nothing success criterion. The reason for this is that there is a target gap between the regularization candidate scoring method based on NAVSIM-v1 and the closed-loop execution of Bench2Drive. The former only supervises the quality of candidates within a limited time range, while repeated planning and execution rely on interactive feedback, controller tracking, and temporal consistency. Therefore, current candidate scoring method may not equally enhance the completion rate or comfort level. Reducing this gap is not the focus of this study and will be addressed in the future.

AuthorsOpenReview ↗

Response to Reviewer meU9 3/3

Q4: NAVSIM-v2 temporal reranking

Using the original evaluator, the complete results are:

MethodImg. backboneNC ↑DAC ↑DDC ↑TLC ↑EP ↑TTC ↑LK ↑HC ↑EC ↑EPDMS ↑
DA-WAM w/o rerankingViT-L98.498.499.199.988.697.997.697.879.687.7
DA-WAM w/ rerankingViT-L99.298.897.899.591.198.793.396.777.189.2

Temporal reranking contributes +1.5 EPDMS. The revised description will distinguish this inference-time contribution from the future-conditioned scoring model.

Reviewer_g97ROpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper argues that a world model helps planning only if its predictions directly affect how each candidate trajectory is scored. When predicted futures are shared or pooled across candidates, the world models is less effective for planning. The paper proposes DA-WAM, where it gives each candidate its own predicted future. In the proposed approach, a V-JEPA 2.1 encoder with LoRA adapters encodes the scene once. Then, a generator proposes 32 candidate trajectories, which a shared predictor turns each candidate into 8 compact future tokens. A scorer receives the scene, the candidate and its future tokens and predicts the utility score. The candidate with the highest utility is selected. It is a two-stage training approach, where in Stage 1, only the candidate closest to the expert trajectory receives a future target. Its tokens are decoded into a dense feature map and regressed onto EMA-encoder features of the frames together with the planning losses. In Stage 2, the encoder and generator are frozen. The predictor and scorer are refined with hard negatives (unsafe trajectories close to the expert), higher loss weights for unsafe candidates and a safe-over-unsafe ranking loss. The paper reports state-of-the-art results on NAVSIM-v1, NAVSIM-v2 and Bench2Drive benchmarks. The ablations compare per-candidate futures with no future, one shared future and prediction of the current features.

Critical strengths
  • S1.The paper makes a clear hypothesis that - each candidate should be scored with its own predicted future. The ablation is Table 4 is very well suited, where the proposed approach is compared against three alternatives including no future prediction, shared future prediction and prediction of the current features. Table 6 further tests on different prediction horizons with action-conditioned future prediction.
  • S2. The proposed approach achieves state-of-the-art results while being efficient during inference by encoding scene only once per set of evaluations.
Critical weaknesses
  • W1. The evidence for the main hypothesis, that per-candidate futures improve selection, is weak. In Table 4, the proposed action-conditioned future improves over no-future prediction by only 0.15 PDMS and over current-feature prediction by 0.21. No error bars are reported to establish statistical significance. Why does the shared-future variant score even worse than no future prediction? This raises questions about the main hypothesis. Further, why does PDMS get worse for horizons longer than 0.5 s in Table 6? This seems at odds with the motivation, since consequences such as collisions unfold over several seconds.
  • W2. Similarly, safety supervision improves PDMS by only 0.22 points and no variance is reported for this comparison. The paper also attributes this Stage 2 improvement to the future latents. However, Stage 2 adds no future supervision. It adds hard negatives, extra weight on unsafe candidates and a ranking loss and it switches the future-prediction loss off. The gain may therefore come from the new training signal alone.
  • W3. The novelty over previous work is not clear. WoTE also predicts a separate future for each candidate and scores each candidate from its future. How is this work different from WoTE in terms of main hypothesis? Additionally, no comparisons are shown for WoTE in table 1 and 3. World4Drive also predicts and scores one future per candidate, questioning the novelty of the hypothesis.
  • W4. The paper lacks presentation/writing quality and some sentences are very cryptically written. For example, the contribution list uses terms that are defined only later, such as ‘expert-matched compact tokens’, ‘trajectory-representation pairing’ and ‘native pose tokens’. The predictor is named differently across the paper. Further the main overview figure (Fig. 2) is hard to understand and does not match the text in Section 3. The per-candidate future path is drawn with training-only arrows, although it is the core of inference. The EMA branch is drawn with training-and-inference arrows, although it is training-only. The EMA encoder is labelled "Frozen", although it is updated by EMA in Stage 1. The dense decoder is missing in the figure, the per-candidate futures are drawn as full spatial maps instead of compact tokens and no arrow links the futures to the scorer.
  • W5. On NAVSIM-v2, DA-WAM uses the corrected evaluator. Table 2 lists SparseDriveV2 at 86.7, which is its score under the original evaluator. Under the corrected evaluator, SparseDriveV2 reports 90.1 in its own paper, so the claimed lead of 1.72 points does not hold. DA-WAM's 89.2 also includes temporal reranking, and no result without it is given.
Questions for authors
  • Q1. Why does one shared future score below no future? Since the gaps are only 0.15 to 0.21 PDMS, could you report Table 4 over several seeds?
  • Q2. Stage 2 changes the training objective. Could you apply Stage 2 to the No-Future variant? Could you also show that the scorer uses the future tokens, for example by swapping future tokens between candidates and checking whether the scores follow the tokens or the trajectories?
  • Q3. The best horizon is 0.5 s and its target window [t, t+0.5 s] includes the current frame (Table 6). How does this fit the motivation of capturing consequences such as collisions over the planning horizon? Could you report future-prediction quality per horizon, compared with simply copying the current features?
  • Q4. WoTE paper predicts a separate future for each candidate, scores each candidate from its own future and uses its world model at inference. What does DA-WAM add beyond the change from BEV to a JEPA latent space?
  • Q5. Please report all methods in Table 2 with the same NAVSIM-v2 evaluator. SparseDriveV2, for example, reports 86.7 with the original and 90.1 with the corrected evaluator.
  • Q6. Add the missing camera-only method, DriveTransformer (ICLR-25), to the Bench2Drive Table 3. They show better performance than the proposed approach.
  • Q7. The proposal generator and its losses are inherited from a base planner that is not named. Which base planner is the proposal generator inherited from and what does it score on NAVSIM on its own?
7 comments
AuthorsOpenReview ↗

Response to Reviewer g97R 1/4

We thank the reviewer for the careful evaluation and constructive feedback. We address the effectiveness of action-conditioned futures, variability across random seeds, and the choice of prediction horizon below.

W1 / Q1: Effectiveness and Robustness of Action-Conditioned Futures

(1) Why does a shared future underperform no future prediction?
A shared future provides the same predicted representation to all candidates, although different candidate trajectories may lead to different future states. It therefore lacks candidate-specific information for distinguishing their consequences and may introduce misleading signals into candidate scoring. This provides a plausible explanation for its lower PDMS compared with the no-future baseline that adding future information is not necessarily beneficial when that information does not reflect the action being evaluated.

To further investigate this difference, we compare the future prediction loss during training:

Epoch01491519
Shared Future0.48670.39060.33630.33850.34470.3503
Action-Conditioned Future (Ours)0.45420.35270.29670.28520.28440.2856

The shared-future loss initially decreases but subsequently plateaus and rises slightly, whereas action-conditioned prediction converges to a lower loss. At epoch 15, action conditioning reduces the prediction loss by approximately 17.5% relative to the shared-future variant. Although this does not by itself establish the cause of the PDMS difference, it is consistent with candidate conditioning making future prediction more informative.

(2) How substantial are the gains?
We acknowledge that the absolute PDMS gains are modest and that the original single-run results alone do not establish their robustness. The relevant baselines already achieve approximately 93–94 PDMS, so our claim is an incremental benefit over a strong planner. To assess whether this benefit is reproducible and attributable to the future prediction target, we provide an equal-capacity comparison and additional results across five random seeds. Even in such a high-performance range, our profile still provides consistent positive returns.

(3) Equal-capacity comparison.
Current-Latent Conditioning (Δ=0\Delta=0) and Action-Conditioned Future Prediction (Δ=0.5\Delta=0.5) use the same architecture, parameter capacity, candidate feature processing, and scorer; only the prediction target changes. The original results are:

ConfigurationWithout Safety RefinementWith Safety Refinement
Current-Latent Conditioning (Δ=0\Delta=0)93.2593.51
Action-Conditioned Future (Δ=0.5\Delta=0.5)93.4693.68
Gain+0.21+0.17

Future prediction improves PDMS under both settings, supporting the benefit of predicting future states beyond that of additional model capacity.

(4) Results across five random seeds.
In response to the reviewer’s request, we repeat the Table 4 experiments across five random seeds:

ConfigurationSafety sup.PDMS ↑NC ↑DAC ↑EP ↑TTC ↑Comfort ↑
No Future PredictionNo93.35 ± 0.020598.64 ± 0.013598.72 ± 0.032190.15 ± 0.022996.19 ± 0.037099.94 ± 0.0213
Shared Global FutureNo92.81 ± 0.011299.01 ± 0.006998.46 ± 0.005888.67 ± 0.008196.55 ± 0.017899.99 ± 0.0000
Current-Latent (Δ=0\Delta=0)No93.24 ± 0.015498.42 ± 0.015898.17 ± 0.008291.36 ± 0.020995.53 ± 0.024499.92 ± 0.0074
Current-Latent (Δ=0\Delta=0)Yes93.50 ± 0.010598.9 ± 0.007498.87 ± 0.003789.84 ± 0.013896.45 ± 0.019799.80 ± 0.0069
Action-Conditioned Future (Δ=0.5\Delta=0.5)No93.45 ± 0.018798.70 ± 0.010198.59 ± 0.025690.98 ± 0.015195.75 ± 0.016999.90 ± 0.0160
Action-Conditioned Future (Δ=0.5\Delta=0.5)Yes93.70 ± 0.007299.12 ± 0.003798.90 ± 0.003789.99 ± 0.011496.81 ± 0.017199.76 ± 0.0074

Across five seeds, action-conditioned future prediction improves mean PDMS over the equal-capacity current-latent baseline by 0.21 without safety supervision and 0.20 with safety supervision. It also exceeds the no-future baseline by 0.10 PDMS without safety supervision, while the shared-future variant remains below that baseline. Our methods improve the safety metrics stabley. These results demonstrate a modest, reproducible benefit from candidate-specific future prediction.

AuthorsOpenReview ↗

Response to Reviewer g97R 2/4

W1 / Q3: Prediction Horizon and Future Prediction Quality

(1) Prediction Quality: Comparison with the Copy-Current Baseline

We evaluated prediction quality against directly copying current features:

Prediction horizonPredicted L1 ↓Copy-current L1 ↓L1 reduction ↑RMSE ↓Cosine distance ↓
0.5 s0.39550.495920.25%0.52190.1392
1.0 s0.45100.626127.97%0.60420.1874
1.5 s0.49480.708430.15%0.67410.2336

Learned prediction reduces L1 error by 20.25%–30.15% relative to Copy-current. This demonstrates predictive improvement over direct feature copying on the evaluated targets.

Longer horizons show larger absolute L1, RMSE, and cosine errors, despite larger relative improvements over Copy-current. A plausible explanation for the best PDMS at 0.5 seconds is therefore a trade-off between predictive informativeness and reliability. This is consistent with the measurements.

W2 / Q2: Separating future supervision from safety refinemen

(1) We agree that the second stage itself introduces useful training signals. It adds hard negative sample sampling, unsafe candidate weights, and pairwise ranking, while disabling the future prediction loss. However, disabling this loss does not remove the candidate-influenced prediction branches from the inference process; they still participate in training. In the second stage, it merely enhances the discriminator's ability to identify safe behaviors.

The No-Future baseline removes the prediction module entirely. To isolate the effect of the predictive target without changing architecture or capacity, we instead use Current-Latent Conditioning (Δ=0\Delta=0). It retains the same candidate-conditioned predictor and scorer, but predicts the current latent rather than a future latent. Both current- and future-latent variants receive the same Stage 2 procedure.

ConfigurationWithout Stage 2With Stage 2Stage 2 gain
Current-Latent Conditioning (Δ=0\Delta=0)93.2593.51+0.26
Action-Conditioned Future (Δ=0.5\Delta=0.5)93.4693.68+0.22
Gain+0.21+0.17—

Stage 2 benefits both variants. Thus, we do not attribute its entire gain to future prediction. Importantly, the future-target model retains a positive measured advantage under identical safety refinement. This is the architecture-matched control for the reviewer's underlying attribution question; we are not suggesting that Stage 2 is technically inapplicable to Current-Latent Conditioning.

(2) We also evaluated safety refinement over five random seeds.

MetricWithout safety: mean ± stdWith safety: mean ± std
PDMS ↑93.4733 ± 0.022593.7039 ± 0.0072
NOC ↑98.8704 ± 0.002399.1166 ± 0.0037
DAC ↑98.5954 ± 0.013898.8984 ± 0.0037
Progress ↑90.4834 ± 0.012589.9922 ± 0.0114
TTC ↑96.3247 ± 0.026496.8105 ± 0.0171
Comfort ↑99.6822 ± 0.009499.7645 ± 0.0074

PDMS increases by 0.2306 points on average, with an improvement in every reported seed. Improvements in NOC, DAC, and TTC accompany a reduction in Progress, consistent with a shift toward safer candidate selection.

(3) Finally, we tested whether the trained scorer uses the candidate-associated latents by changing their pairing at evaluation:

Pairing conditionPDMS ↑Δ PDMSFuture latent L1 ↓
Original pairing93.6806—0.336550
Joint permutation of trajectories and latents93.68060.00000.336550
Latent swapping between similar trajectories91.0207−2.65990.337297
Random latent swapping, five-trial mean90.5967−3.08390.342251
Latent swapping between dissimilar trajectories88.5346−5.14600.346909

The joint-permutation control leaves performance unchanged, whereas breaking the pairing reduces PDMS by 2.66–5.15 points. This supports the importance of trajectory–latent correspondence.

AuthorsOpenReview ↗

Response to Reviewer g97R 3/4

W3 / Q4 /Q6/ Q7: Comparison on other methods

We thank the reviewer for highlighting the need to clarify our relationship to prior work and identify the base planner.

(1) Main contribution and comparison with WoTE. As acknowledged in line 44 of the original manuscript, WoTE predicts candidate-specific future BEV states, evaluates candidates using their respective futures, and employs its world model at inference. Over-reliance on BEV can be limited by the labeling accuracy of BEV itself, thereby restricting the upper limit of the planner. DA-WAM’s contribution therefore extends beyond candidate-specific future prediction. It combines JEPA-based predictive grounding, explicit supervision of candidate-level planning utility, and safety-focused refinement to improve trajectory selection. Beyond replacing BEV features with compact JEPA latents, DA-WAM introduces a dedicated predictive training mechanism.

(2) Predictive supervision and comparison with World4Drive. As acknowledged in line 45 of the original manuscript, World4Drive also predicts multiple intention-conditioned latent futures. Its selector is trained to select the candidate whose predicted future features best match the observed future representation. DA-WAM instead learns to predict planning factors and trajectory utility from each candidate and its associated predictive latent, directly aligning candidate evaluation with driving quality. Since World4Drive also incorporates trajectory imitation supervision, the distinction lies specifically in the selector’s training objective, rather than the presence or absence of planning supervision.

DA-WAM further maintains a LoRA-adapted V-JEPA online encoder and a separate EMA target encoder during joint training. Dense future-feature regression is applied to the expert-matched candidate to ground its representation in observed evolution. At inference, the scorer directly consumes compact candidate-conditioned latents, and the dense decoder is removed. This design separates predictive grounding from trajectory scoring while allowing both objectives to shape the shared representation during Stage 1.

For the comparison, we also list the score of these methods:

MethodInputsNC ↑DAC ↑TTC ↑Comfort ↑EP ↑PDMS ↑
World4DriveCameras97.4094.3092.80100.0079.9085.10
WoTECameras + LiDAR98.5096.8094.9099.9081.9088.30
iPadCameras98.6098.3094.90100.0088.0091.70
DA-WAM (five-seed mean)Cameras99.1298.9096.8199.7689.9993.70

DA-WAM achieves 93.70 PDMS, exceeding the published results of World4Drive and WoTE by 8.60 and 5.40 points, respectively.

(3) Safety-focused candidate discrimination. DA-WAM introduces expert-proximate hard negatives and safety-ranking refinement to distinguish geometrically similar trajectories with different safety outcomes. This explicitly trains the scorer to recognize safety differences that trajectory similarity alone may not capture. The attention visualizations in Figure 6 show increased responses around interacting vehicles and traffic signals after refinement, providing qualitative supporting evidence. Our component ablations and five-seed results provide the stronger quantitative evidence for the effectiveness of this design.

(4) Base planner and standalone performance. Our proposal generator and its associated losses are inherited from iPad, which reports 91.70 PDMS on NAVSIM-v1 as a standalone planner. DA-WAM achieves 93.70 PDMS averaged over five seeds, improving upon iPad’s published result by 2.00 points. We have already reported iPad’s performance in line 302 of the original manuscript. We will further clarify that iPad is our base planner and explicitly identify the inherited proposal generator and associated losses in the revised manuscript.

(5) We agree that DriveTransformer and WoTE should be included rather than omitted. Their published results provide the following context:

MethodAvg. L2 (m) ↓Efficiency ↑Comfortness ↑Success rate (%) ↑Driving score ↑
SeerDrive0.66——30.1758.32
WoTE———31.3661.71
DriveTransformer-Large0.62100.6420.7835.0163.46
DA-WAM0.95140.4014.6424.6659.04

DA-WAM achieves a competitive driving score of 59.04, within 4.33% of WoTE and 6.97% of DriveTransformer-Large, while delivering 39.51% higher efficiency than DriveTransformer-Large (140.40 vs. 100.64). These results demonstrate a substantial efficiency advantage alongside a relatively modest driving-score gap, supporting the competitiveness of our approach on Bench2Drive. The reason to this gap is a mismatch between the scorer’s candidate evaluations and the outcomes realized during closed-loop execution.

Comments on the paper

2 comments

Thank you so much for your effort during the review process and for raising your score! Your valuable feedback has greatly helped us improve the quality of our paper. We’ll incorporate the points raised in the discussion into the final version. Thanks again!

AuthorsOpenReview ↗

General Response

We sincerely thank you and all reviewers for the time and effort spent evaluating our submission and for the constructive feedback.

Our discussions helped us articulate our contribution more clearly. A long-standing goal in autonomous driving is to find compact scene representations that retain the information most relevant to driving decisions. Semantic BEV maps and occupancy grids pursue this goal through hand-designed representations. DA-WAM learns a compact, decision-aligned future representation for each candidate trajectory directly from data. During training, future-feature supervision grounds these representations in observed scene evolution, while candidate-level planning objectives shape what they retain for trajectory evaluation.

Summary of revisions:

1. For Reviewers g97R and iQyG, we improved Figure 1 and clarified our contribution relative to WoTE and World4Drive in the Introduction and Related Work. To our knowledge, DA-WAM is the first driving world model to jointly train compact, candidate-conditioned future representations with both JEPA future-feature supervision and explicit per-candidate planning-metric targets.

2. For Reviewers g97R, meU9, and iQyG, we added an equal-capacity comparison in Table 4 and a candidate–future swapping experiment in Appendix Table 9. These experiments test the benefit of predicting future features and whether the scorer uses each candidate’s corresponding prediction.

3. For Reviewers g97R and meU9, we clarified how the architecture supports safety-focused post-training, isolated the contributions of the three Stage 2 training components in Appendix Table 10, and expanded the visual and quantitative analyses in Fig. 6 and Appendix Figs. 12–14. The revisions are highlighted in blue.

General responses to common questions:

GR1. How do candidate-conditioned future representations contribute to trajectory evaluation?

With the architecture, parameter count, and scorer held fixed, changing the prediction target from current features to future features improves PDMS both before and after Stage 2.

Prediction targetBefore Stage 2After Stage 2
Current features93.2593.51
Future features93.4693.68
Gain from future prediction+0.21+0.17

Three analyses clarify how the model uses these representations.

  • Candidate–future swapping (Appendix Table 9). Reordering candidates together with their representations preserved PDMS exactly. Swapping only the representations reduced PDMS by 2.66 points for similar trajectories, 3.08 for random swaps, and 5.15 for dissimilar trajectories. Trajectory selection therefore depends on the correct candidate–future pairing.
  • Within-scene correlations (Appendix Fig. 14). Across 496 candidate pairs per scene, the mean within-scene Spearman correlation was 0.648 between trajectory displacement and representation distance, and 0.537 between representation distance and absolute predicted score difference.
  • Qualitative analysis (Fig. 6 and Appendix Figs. 12–13). In vehicle following, lane changing versus stopping, and driving through a bend, different candidate trajectories produced distinct future-feature predictions from the same observation.

GR2. How is DA-WAM tailored to Stage 2, and what does this stage improve?

We tailor DA-WAM to counterfactual supervision in Stage 2 by using explicit trajectory coordinates as both the generator’s output and the trajectory input to the future prediction branch. This design lets us supply an external counterfactual trajectory directly to the frozen predictor and obtain its own future representation. Stage 2 then trains only the trajectory scorer to distinguish geometrically similar candidates with different safety outcomes. Hard-negative sampling and unsafe-candidate weighting individually improve PDMS by 0.19 and 0.13. Ranking loss alone increases progress by 1.33 points but reduces PDMS by 0.84. The full configuration achieves the highest PDMS, achieving a balance between safety and progress.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.