acceptodds
Under review as a conference paper at ICLR 2027

Learn the Operator, Not the Future: State-Anchored Distillation of World Action Model Dynamics for Autonomous Driving

Abstract

World action models(WAM) offer end-to-end driving a glimpse of the future, yet directly feeding WAM-predicted features into a planner suffers from feature misalignment, error accumulation in closed-loop rollouts, and high runtime cost. Moreover, generic video WAMs model how scenes evolve but are not explicitly aligned with driving behavior. We propose state-anchored transition transfer, which turns the WAM from a runtime feature provider into a training-time source of transferable transition knowledge. A compact transition operator learns to transform current scene features toward future WAM representations and modulates the planner through a lightweight FiLM-style mechanism, while planning remains anchored to fresh observations. Importantly, the approach requires no future-frame observations during training and relies solely on the pretrained WAM to provide future representations, while the WAM is completely removed at inference. This keeps hallucinated futures off the control path and eliminates the computational cost of WAM deployment. To further align the learned transitions with driving behavior, we use a rule-supervised scorer trained with a metric simulator to guide transition learning. On the NAVSIM-v2 navhard benchmark, our method improves the baseline EPDMS by 13.7% (from 48.3 to 54.9), and under zero-shot closed-loop evaluation on HUGSIM it further gains 25.4% in driving score, achieving state-of-the-art performance with negligible inference overhead.

Primary area:
applications to robotics, autonomy, planning
Keywords:
world-action model, AD, End-to-End Planning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_cMYzOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a world-model-guided representation learning framework for end-to-end autonomous driving. It introduces a lightweight transition operator that distills future dynamics knowledge from a pretrained video world model (Wan-2.2 5B) into compact semantic transition factors, which are used to modulate current scene representations. The method further incorporates driving-rule knowledge from a trajectory scorer through physics tokens and representation alignment. During inference, the world model and future prediction head are removed, maintaining an efficient planning pipeline. Experiments on NAVSIM and HUGSIM demonstrate improvements in planning performance, particularly in challenging driving scenarios.

Critical strengths
  1. WAM Distillation and State-Anchored Planning. The paper presents an interesting approach to distilling future dynamics into compact transition factors that enhance current scene representations, enabling efficient planning without requiring future prediction at inference time.
  2. Strong performance. The proposed method achieves strong results on NAVSIM and HUGSIM, demonstrating significant improvements over the baseline, particularly in challenging driving scenarios.
Critical weaknesses

While the overall performance is strong, the paper would benefit from more explicit evidence and explanations of the proposed mechanisms, particularly regarding what the learned representations capture and why these designs are effective.

  1. Insufficient Evidence for Ego-Motion Factorization. The paper claims that ego-conditioned queries help disentangle ego-induced changes from surrounding scene dynamics. However, the experiments mainly demonstrate improvements in planning performance, without directly verifying whether the learned transition factors achieve such disentanglement. More analysis of the learned representations would help support this claim.
  2. Limited Justification for Driving-Knowledge Completion. The motivation and effectiveness of this component need further clarification. In particular, why should aligning with the scorer's intermediate representations lead to acquiring driving-rule or physical knowledge? What specific knowledge is transferred, and can the authors provide more direct evidence of this? Moreover, the relatively modest performance gain raises questions about the actual contribution of this component.
Questions for authors

See weaknesses.

1 comment

We thank the reviewer for the thoughtful and constructive review, and for recognizing the value of distilling future dynamics into compact transition factors and the strong results on NAVSIM and HUGSIM. We respond to each point below, and the new results have been incorporated into the revised manuscript.

W1 → A1

We agree that verifying ego-motion disentanglement requires direct evidence from the learned factors. We condition the pooling queries on ego status to give the operator an explicit cue about the vehicle’s motion when extracting transition factors. In Table 4(a), adding ego conditioning before rule injection improves navhard EPDMS from 52.3 to 54.4, a gain of 2.1 points.

We also examined the learned FiLM coefficients across 5,912 evaluated scenes. Fewer than 0.66% of the token-channel scale coefficients 1+γ1+\gamma were within 0.01 of zero, based on a histogram upper bound. This characterizes how often the operator nearly suppresses the direct contribution of an observed feature.

We have revised the Introduction and Conclusion to remove the overstated claim. The revised text states the intent of the design more precisely. We provide ego status to the pooling queries so that the operator can take the ego vehicle’s motion into account when extracting transition factors.

W2 → A2

Thank you for this thoughtful question. Our motivation is to complement the world model’s prediction of how a scene may evolve with information about how a candidate trajectory should be evaluated in that scene. The scorer ranks candidate trajectories and is supervised by simulator labels for six criteria: at-fault collision avoidance, drivable-area compliance, driving-direction compliance, time-to-collision, route progress, and comfort. These prediction heads share a trajectory-specific representation. During training, we use each candidate to query the physics tokens and align the resulting feature with the scorer’s detached representation for that candidate. This gives the physics tokens a training signal tied to the criteria the scorer uses to evaluate trajectories. The tokens then influence planning through the modulation of current scene features.

\text{Metric} & \text{w/o rule injection} & \text{w/ rule injection} \\\\ \hline \text{No at-fault collisions} & 89.1 & 90.4 \\\\ \text{Drivable-area compliance} & 86.8 & 88.4 \\\\ \text{Driving-direction compliance} & 92.9 & 93.6 \\\\ \text{Ego progress} & 84.4 & 85.3 \\\\ \text{Time-to-collision within bound} & 87.6 & 87.9 \\\\ \text{Lane keeping} & 51.6 & 53.3 \\\\ \text{History comfort} & 98.4 & 98.8 \\\\ \hline \text{Overall EPDMS} & 54.4 & \mathbf{54.9} \end{array}$$ In the navhard module ablation, enabling rule injection with the one-way mask (aboved table) raises EPDMS from 54.4 to 54.9. The Stage 2 breakdown shows gains in no-at-fault collisions (89.1 to 90.4), drivable-area compliance (86.8 to 88.4), driving-direction compliance (92.9 to 93.6), and lane keeping (51.6 to 53.3). These improvements show where the modest aggregate gain appears in driving behavior. In a separate architecture-matched experiment, we retain the original scorer and all planner modules and vary only the scorer-feature alignment loss. Enabling this loss raises EPDMS from 52.4 to 54.9, isolating the contribution of the supervision at fixed model capacity. The alignment head runs only during training, and reverse injection reuses the existing scorer at inference, so it requires no additional scoring pass. This makes reverse injection a cost-effective way to improve collision avoidance, drivable-area compliance, and lane keeping.
Reviewer_RogkOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes state-anchored transition transfer for incorporating pretrained world-model knowledge into an end-to-end driving planner without deploying the world model at inference. Building on DrivoR, it uses offline future representations from a frozen video model to supervise compact, ego-conditioned transition factors. These factors, together with tokens distilled from a rule-supervised trajectory scorer, modulate current scene features through a FiLM-style mechanism. The world model and auxiliary prediction heads are removed at deployment. Evaluation on NAVSIM and zero-shot HUGSIM demonstrates substantial improvements on navhard and aggregate closed-loop driving scores, supported by component ablations and comparisons with direct future-feature consumption.

Critical strengths
  1. A practical architecture for transferring world-model knowledge into a deployable planner. The method addresses an important deployment challenge: benefiting from a large pretrained world model while avoiding its runtime execution. Using compact transition factors to modulate current observations is a coherent architectural choice that preserves a lightweight inference pathway. This makes the approach practically relevant, although the precise additional overhead remains to be measured.

  2. Substantial improvements over a strong baseline on challenging evaluations. The method improves navhard EPDMS from 48.3 to 54.9 and average HUGSIM HD-Score from 27.9 to 35.0. The combination of pseudo-closed-loop evaluation and zero-shot closed-loop testing provides meaningful evidence of its utility. Comparisons against future-feature replacement and concatenation also support the effectiveness of the proposed feature-consumption design.

Critical weaknesses
  1. The aggregate closed-loop improvement does not extend consistently to the most difficult scenarios. On HUGSIM Extreme, HD-Score drops from 14.3 to 9.3 and route completion decreases from 30.3 to 20.0 relative to DrivoR, despite substantial gains at Medium difficulty. These regressions are consequential because robustness under difficult interactions is central to the motivation for incorporating world-model knowledge. Without a failure analysis and information about evaluation variability, the scope of the claimed generalization benefit remains unclear.

  2. The source of the substantial performance gain is not sufficiently isolated. The component ablations are useful, but introducing the distilled operator changes both the architecture and its supervision. An architecture-matched control retaining the modulation module while removing world-model distillation is missing. Consequently, the experiments do not clearly distinguish the contribution of transferred dynamics from additional capacity or auxiliary representation learning. This limits confidence in the paper’s central mechanistic explanation, even though the overall performance improvement is convincing.

Questions for authors
  1. Clarification and failure analysis: What explains the substantial regression on HUGSIM Extreme despite the improvement in the aggregate score? Please provide a breakdown of the dominant failure modes, the number of evaluated scenarios, and the variability across repeated evaluations. This would help establish the conditions under which the proposed approach improves or harms closed-loop behavior.

  2. Additional validation of the current mechanism: Can you provide an architecture-matched ablation that retains the pooling, ego conditioning, and FiLM modulation but removes the world-model distillation loss? A complementary control using static rather than future teacher representations would further clarify whether the gains specifically depend on transferred temporal knowledge. Please also report variability across training seeds for the key comparisons.

  3. Efficiency clarification: What are the measured end-to-end inference latency, parameter count, and computational overhead relative to DrivoR under the same hardware and inference settings? Removing the world model clearly eliminates its runtime cost, but measurements are needed to establish whether the remaining transition module introduces negligible additional latency.

4 comments
AuthorsOpenReview ↗

response-part1

We thank the reviewer for the careful and constructive review, and for recognizing the practicality of the deployment design and the strength of the improvements over a strong baseline. Below we respond to each point. The new results have been incorporated into the revised manuscript.

W1 & Q1 → A1

HUGSIM. Thank you for highlighting the regression on HUGSIM Extreme.

HUGSIM contains 436 scenarios: 80 Easy, 157 Medium, 96 Hard, and 103 Extreme. The results in the submission came from an earlier training run. We re-trained our model; it reaches 55.7 EPDMS on NAVSIM-v2 navhard and 33.4 average HD-Score on HUGSIM (27.9 for DrivoR). The table below reports zero-shot closed-loop results for both runs.

Relative to DrivoR, the re-trained model gains most at Medium (RC +10.4, HD-Score +13.7), and at Extreme reaches 15.1 HD-Score (+0.8) with 28.1 RC (-2.2). The HD-Score regression no longer appears, and we do not claim an improvement at Extreme difficulty. We will release the latest model weights as open source.

& & & \text{RC}\uparrow & & & & & \text{HD-Score}\uparrow & & \\\\ \text{Method} & \text{E} & \text{M} & \text{H} & \text{X} & \text{Avg.} & \text{E} & \text{M} & \text{H} & \text{X} & \text{Avg.} \\\\ \hline \text{DrivoR} & 78.6 & 42.7 & 29.3 & 30.3 & 43.4 & 68.9 & 24.6 & 13.5 & 14.3 & 27.9 \\\\ \text{GTRS-Dense} & 64.2 & 50.0 & 20.7 & 22.3 & 38.0 & 55.5 & 39.0 & 11.7 & 14.3 & 28.6 \\\\ \text{Latent-WAM} & 84.2 & 42.5 & 30.6 & 35.5 & 45.9 & 72.5 & 24.0 & 12.2 & 18.1 & 28.9 \\\\ \hline \text{Ours (submitted)} & 79.8 & 56.3 & 27.9 & 20.0 & 45.5 & 72.5 & 44.5 & 17.6 & 9.3 & 35.0 \\\\ \text{Ours (re-trained)} & 76.2 & 53.1 & 28.6 & 28.1 & 45.8 & 68.9 & 38.3 & 16.4 & 15.1 & 33.4 \\\\ \end{array}$$ ### W2 → A2 We agree with your assessment. To address this, we trained architecture-matched controls on NAVSIM-v2 navhard. Every variant retains the same ego-conditioned pooling, semantic and physics token branches, and FiLM modulation. We keep the training budget fixed and vary only the weights of the two distillation losses. The full model achieves 54.9 EPDMS with $(\lambda_d, \lambda_p)=(1.0,0.5)$. Setting $\lambda_d=0$ while retaining scorer-feature distillation reduces EPDMS to 50.9, a 4.0-point drop that isolates the contribution of world-model supervision in the full architecture. Setting $\lambda_p=0$ while retaining world-model supervision yields 52.4, a 2.5-point drop. With both losses disabled, the same architecture reaches 49.2, compared with 48.3 for the original DrivoR baseline. Restoring both distillation losses therefore yields a 5.7-point gain at fixed architecture and parameter count. These controls quantify how much the training supervision contributes when the modulation module is held fixed. We have added the results to Table 6 in Appendix B of the revised manuscript. $$\begin{array}{ccccc} \text{WAM loss} & \text{Scorer distillation} & \lambda_d & \lambda_p & \text{EPDMS}\uparrow \\\\ \hline \times & \checkmark & 0 & 0.5 & 50.9 \\\\ \checkmark & \times & 1.0 & 0 & 52.4 \\\\ \times & \times & 0 & 0 & 49.2 \\\\ \checkmark & \checkmark & 0.5 & 0.5 & 51.0 \\\\ \checkmark & \checkmark & 1.5 & 0.5 & 53.4 \\\\ \checkmark & \checkmark & 2.0 & 0.5 & 53.2 \\\\ \checkmark & \checkmark & 1.0 & 0.25 & 54.3 \\\\ \checkmark & \checkmark & 1.0 & 1.0 & 53.7 \\\\ \hline \checkmark & \checkmark & 1.0 & 0.5 & \mathbf{54.9} \end{array}$$
AuthorsOpenReview ↗

response-part2

Q2 → A3

We ran an architecture-matched ablation on NAVSIM-v2 navhard that retains the pooling module, ego conditioning, semantic and physics token branches, and FiLM modulation. We held the training budget fixed. The full model achieves 54.9 EPDMS with (λd,λp)=(1.0,0.5)(\lambda_d, \lambda_p)=(1.0,0.5). Setting λd=0\lambda_d=0 while retaining scorer-feature distillation reduces EPDMS to 50.9, a 4.0-point drop at fixed planner capacity. Removing scorer-feature distillation alone yields 52.4, and removing both distillation losses yields 49.2. We report these loss-weight ablations in Table 6 of Appendix B.

\text{WAM loss} & \text{Scorer distillation} & \lambda_d & \lambda_p & \text{EPDMS}\uparrow \\\\ \hline \times & \checkmark & 0 & 0.5 & 50.9 \\\\ \checkmark & \times & 1.0 & 0 & 52.4 \\\\ \times & \times & 0 & 0 & 49.2 \\\\ \hline \text{Static Wan2.2 targets} & \checkmark & 1.0 & 0.5 & 53.8 \\\\ \checkmark & \checkmark & 1.0 & 0.5 & \mathbf{54.9} \end{array}$$ For the static control, we kept both loss weights fixed and replaced the future DiT targets with a cached Wan2.2 encoder feature from the current front-camera image. We used the same static target at each of the eight student steps. This control achieves 53.8 EPDMS, 2.9 points above the variant without Wan2.2 supervision. Future-feature distillation adds 1.1 points over the static control. Because the two targets also come from different layers of Wan2.2, we interpret this gain as evidence for the effectiveness of our future-feature targets in this setting. Based your suggestion, we implement 2nd Run with different seed. The below table reports the Stage 1 and Stage 2 metrics for each run, which indicates the obvious gains of our methods. $$\begin{array}{lc|ccccccccc|c} \text{Training run} & \text{Stage} & \text{NC} & \text{DAC} & \text{DDC} & \text{TLC} & \text{EP} & \text{TTC} & \text{LK} & \text{HC} & \text{EC} & \text{EPDMS}\uparrow \\\\ \hline \text{1st Run (submitted)} & \text{Stage 1} & 99.6 & 96.4 & 99.6 & 100 & 90.7 & 99.1 & 93.1 & 97.6 & 70.2 & 54.9 \\\\ & \text{Stage 2} & 90.5 & 88.4 & 93.6 & 98.5 & 85.3 & 87.9 & 53.3 & 98.9 & 75.9 & \\\\ \hline \text{2nd Run (re-trained)} & \text{Stage 1} & 98.6 & 96.9 & 99.2 & 99.8 & 92.9 & 98.4 & 93.3 & 97.8 & 71.6 & 55.7 \\\\ & \text{Stage 2} & 89.2 & 88.7 & 92.9 & 98.3 & 84.9 & 87.6 & 51.8 & 97.7 & 77.3 & \end{array}$$ ### Q3 → A4 We measured the inference cost of the deployed planner, including the transition operator, against DrivoR under identical settings. We ran five paired benchmarks on an NVIDIA H20 with batch size 1 and automatic mixed precision disabled. We report the mean ± sample standard deviation across the five run means. From prepared CPU inputs to the returned trajectory, latency was 134.48 ± 0.19 ms for DrivoR and 135.01 ± 0.07 ms for our method, an average increase of 0.53 ms (0.39%). This measurement includes CPU-to-GPU input transfer, the planner forward pass, and GPU-to-CPU transfer of the predicted trajectory. Both methods use synthetic inputs, and the timing starts after image preprocessing. For the model forward pass alone, latency was 131.25 ± 0.30 ms for DrivoR and 132.94 ± 2.11 ms for our method, an average increase of 1.69 ms (1.29%). $$\begin{array}{lrrr} \text{Metric} & \text{DrivoR} & \text{Ours} & \text{Increase} \\\\ \hline \text{Inference parameters (M)} & 40.788 & 43.365 & +6.32\\% \\\\ \text{Model forward latency (ms)} & 131.25 \pm 0.30 & 132.94 \pm 2.11 & +1.29\\% \\\\ \text{Peak allocated GPU memory (MiB)} & 506.239 & 516.072 & +1.94\\% \\\\ \text{Estimated computation (GFLOPs)} & 1856.154 & 1856.268 & +0.0062\\% \\\\ \hline \text{EPDMS (NAVSIM-v2 navhard)}\uparrow & 48.3 & \ 54.9 & +13.7\\% \end{array}$$ The deployed parameter count increases from 40.788M to 43.365M (+6.32%), while estimated computation increases from 1856.154 to 1856.268 GFLOPs per sample, a difference of 0.114 GFLOPs (approximately 0.0062%). Peak allocated GPU memory increases by 1.94%. We report the forward timings, parameter counts, memory usage, and compute estimates in Table 7 of Appendix C of the revised manuscript.
Reviewer_RogkOpenReview ↗

Thanks for Comments

Thank you for the additional experiments. The architecture-matched ablation addresses my main concern: removing world-model supervision while keeping the planner and scorer distillation fixed reduces navhard EPDMS from 54.9 to 50.9. The future targets add 1.1 points over the static control, which is encouraging, although that comparison also changes the feature source. The H20 timing helps quantify the planner overhead. On HUGSIM Extreme, the retrained model no longer trails DrivoR on HD-Score, but route completion remains lower. I would keep that limitation explicit in the paper.

AuthorsOpenReview ↗

Thanks for discussion

Thank you for the constructive discussion. The revised version will state them explicitly.

Reviewer_Smm9OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper transfers a frozen video model's predicted features into a compact driving planner. Ego-conditioned bottleneck tokens modulate current scene features, while a second branch receives supervision from the planner's rule-based scorer. The video model and prediction head are removed at inference. Results improve DrivoR from 48.3 to 54.9 EPDMS on navhard and from 27.9 to 35.0 HD-Score on HUGSIM.

Critical strengths
  1. The deployment design is practical: the large video teacher is used offline, and the planner retains a compact representation at inference. Avoiding observed future images for the distillation targets is also useful.
  2. The navhard improvement is substantial, and the zero-shot HUGSIM evaluation adds evidence beyond a single offline benchmark. Appendix A also includes direct-future replacement and concatenation controls, which makes the feature-consumption comparison more informative.
Critical weaknesses
  1. The experiments do not isolate transferred dynamics from the added architecture. Table 4 introduces the operator together with WAM supervision; there is no otherwise identical modulator trained without that supervision or with current-image targets. Appendix A compares different downstream token counts, as the authors acknowledge. Thus, the results establish a useful adapter, but do not yet establish that learning a transition operator is responsible for its advantage.
  2. The claim that imperfect dynamics "cannot replace" observations is not supported by Eq. (2): unconstrained scale and shift can suppress or dominate the original features. Removing explicit future tokens does not establish immunity to errors in distilled predictions. Robustness also needs closer examination: on HUGSIM's extreme split, RC falls from 30.3 to 20.0 and HD-Score from 14.3 to 9.3 relative to DrivoR (Table 3). The cause of this regression remains unclear.
  3. Several citation and reporting errors need correction. Wan's cited arXiv identifier has a different title, DriveWorld-VLA is duplicated under two first authors, and Table 1 misdefines TLC and HC. More substantively, the already-cited DriveFuture reports 55.5 navhard EPDMS, yet is absent from Table 1. Please reconcile protocols and qualify the state-of-the-art claim. Calling the image-conditioned Wan teacher an action model also requires clarification.
Questions for authors
  1. With the same operator and training budget, how do removing WAM distillation and replacing future targets with current-image targets affect performance?
  2. Are the modulation parameters bounded or otherwise constrained? Can an inaccurate-teacher control, together with an analysis of the extreme HUGSIM failures, substantiate the grounding claim?
  3. Which teacher layer, denoising step, prompts, and frame timestamps define Eq. (3)? Please also report measured inference latency and repeated-run variation for the principal comparisons.
6 comments
AuthorsOpenReview ↗

response-part1

We thank the reviewer for the detailed and constructive review, and for recognizing the practical deployment design and the zero-shot HUGSIM evaluation. We respond to each point below, and the new results and corrections have been incorporated into the revised manuscript.

W1 & Q1 → A1

We agree that the effect of WAM supervision was not isolated. In Table 4 the operator and its distillation supervision are introduced together, and in Appendix A the future-replacement and concatenation variants also change the number of tokens seen by the decoder. We therefore ran controls on NAVSIM-v2 navhard with the planner architecture and training budget held fixed. All variants retain ego-conditioned pooling, both token branches, and FiLM modulation.

Removing Wan2.2 supervision. The full model achieves 54.9 EPDMS. Removing WAM distillation while keeping the architecture unchanged reduces EPDMS to 50.9, a 4.0-point drop at fixed planner capacity. This shows that the gain over the DrivoR baseline (48.3) cannot be attributed to the added modules alone.

Current-image targets. We keep both loss weights at their default values and replace the future targets with a cached Wan2.2 encoder feature from the current front-camera image, used as the same target at each of the eight student steps. This control reaches 53.8 EPDMS, 2.9 points above the variant without Wan2.2 supervision. Future-feature targets add a further 1.1 points. Because the two targets are taken from different layers of Wan2.2, we interpret this difference as evidence for the effectiveness of our future-feature targets in this setting, not as a layer-controlled comparison.

\text{WAM loss} & \text{Scorer distillation} & \lambda_d & \lambda_p & \text{EPDMS}\uparrow \\\\ \hline \times & \checkmark & 0 & 0.5 & 50.9 \\\\ \checkmark & \times & 1.0 & 0 & 52.4 \\\\ \times & \times & 0 & 0 & 49.2 \\\\ \hline \checkmark & \checkmark & 0.5 & 0.5 & 51.0 \\\\ \checkmark & \checkmark & 1.5 & 0.5 & 53.4 \\\\ \checkmark & \checkmark & 2.0 & 0.5 & 53.2 \\\\ \checkmark & \checkmark & 1.0 & 0.25 & 54.3 \\\\ \checkmark & \checkmark & 1.0 & 1.0 & 53.7 \\\\ \hline \text{Static Wan2.2 targets} & \checkmark & 1.0 & 0.5 & 53.8 \\\\ \checkmark & \checkmark & 1.0 & 0.5 & \mathbf{54.9} \end{array}$$ ### W2 & Q2 → A2 Thank you for these questions. **Modulation parameters.** The FiLM coefficients in Eq. (2) are not bounded or otherwise constrained, so Eq. (2) does not by itself prevent the scale from suppressing the observed features, and our statement that imperfect dynamics "cannot replace" them was too strong. We measured the learned coefficients across 5,912 evaluated scenes. The mean and standard deviation were 0.698 and 1.803 for $\gamma$, and 0.057 and 0.733 for $\beta$. Fewer than 0.66% of token-channel scale coefficients $1+\gamma$ were within 0.01 of zero, based on an upper bound from the histogram, so near-cancellation of the direct observation term was uncommon. We have revised Section 3.2 to describe the modulation of current scene features without claiming that it guarantees their preservation or immunity to errors in the distilled predictions, and we have changed the corresponding wording in the Introduction. **HUGSIM.** The Extreme falls you noted (RC 30.3 to 20.0, HD-Score 14.3 to 9.3) came from an earlier training run. After re-training, the model reaches 55.7 EPDMS on NAVSIM-v2 navhard and 33.4 average HD-Score on HUGSIM (27.9 for DrivoR). At Extreme (103 of 436 scenarios), HD-Score is 15.1 versus 14.3 for DrivoR, while RC is 28.1 versus 30.3. The HD-Score falls no longer appears, and we do not claim an improvement at Extreme difficulty. We will release the latest model weights as open source. $$\begin{array}{l|ccccc|ccccc} & & & \text{RC}\uparrow & & & & & \text{HD-Score}\uparrow & & \\\\ \text{Method} & \text{E} & \text{M} & \text{H} & \text{X} & \text{Avg.} & \text{E} & \text{M} & \text{H} & \text{X} & \text{Avg.} \\\\ \hline \text{DrivoR} & 78.6 & 42.7 & 29.3 & 30.3 & 43.4 & 68.9 & 24.6 & 13.5 & 14.3 & 27.9 \\\\ \text{GTRS-Dense} & 64.2 & 50.0 & 20.7 & 22.3 & 38.0 & 55.5 & 39.0 & 11.7 & 14.3 & 28.6 \\\\ \text{Latent-WAM} & 84.2 & 42.5 & 30.6 & 35.5 & 45.9 & 72.5 & 24.0 & 12.2 & 18.1 & 28.9 \\\\ \hline \text{Ours (submitted)} & 79.8 & 56.3 & 27.9 & 20.0 & 45.5 & 72.5 & 44.5 & 17.6 & 9.3 & 35.0 \\\\ \text{Ours (re-trained)} & 76.2 & 53.1 & 28.6 & 28.1 & 45.8 & 68.9 & 38.3 & 16.4 & 15.1 & 33.4 \\\\ \end{array}$$
AuthorsOpenReview ↗

response-part2

W3 → A3

Thank you for pointing these out.

Citations and definitions. We corrected the title associated with arXiv:2503.20314 and added a separate citation to the official Wan2.2-TI2V-5B model card. We consolidated the duplicate DriveWorld-VLA references into a single entry under Jia et al. (arXiv:2602.06521) and updated its citation in Table 2. In the caption of Table 1, we corrected TLC to Traffic Light Compliance and HC to History Comfort. We also revised the method section to describe our teacher explicitly as a frozen, image-conditioned video world model.

DriveFuture and the state-of-the-art claim. We have re-trained our model, and the new model reaches 55.7 EPDMS on NAVSIM-v2 navhard, exceeding DriveFuture's reported 55.5. The table below compares the two methods on NAVSIM-v2 navhard and NAVSIM-v1 navtest.

BenchmarkMetricDriveFutureOurs
NAVSIM-v2 navhardEPDMS55.555.7
NAVSIM-v1 navtestPDMS90.793.8
SettingFuture-frame GT used in trainingYesNo
External GTRS-Dense scorerYesNo
World-model trainingJoint with plannerNone (frozen, cached features)

Compared with DriveFuture, our method uses no future frames, no external scorer, and no WAM training: the frozen WAM only provides cached features offline and is removed at inference. Our goal is therefore not to maximize the leaderboard score but to show how much a pretrained WAM can contribute under this lightweight setting. We will also release the latest model weights as open source.

Q3 → A4

Thank you for this thoughtful question.

Eq.3 settings. To construct the teacher targets in Eq. (3), we extract activations from the 16th DiT block of frozen Wan2.2-TI2V-5B (zero-based index 15 of 30) at the final denoising step (step 16 of 16), using Flow-UniPC with a sampling shift of 5.0. We use an empty text prompt, disable classifier-free guidance, and condition the teacher on the current front-camera image resized to 320 × 160. The 33-frame sequence yields nine latent slices with a temporal stride of four. After discarding the input-anchored slice at frame 0, the eight retained slices have nominal frame offsets of 4, 8, 12, 16, 20, 24, 28, and 32, corresponding to approximately 0.167, 0.333, 0.500, 0.667, 0.833, 1.000, 1.167, and 1.333 s after the input frame at 24 fps. We spatially average-pool these slices and cache the resulting 8 × 3072 tensor offline.

Inference latency. We repeated the paired latency benchmark five times on an NVIDIA H20 with batch size one and automatic mixed precision disabled. The wall-clock model forward latency was 131.25 ± 0.30 ms for DrivoR and 132.94 ± 2.11 ms for our method. We report the mean and sample standard deviation across the five run means. The average forward overhead was 1.69 ms (1.29%). These measurements use synthetic inputs and exclude image preprocessing.

\text{Metric} & \text{DrivoR} & \text{Ours} & \text{Increase} \\\\ \hline \text{Model forward (ms)} & 131.25 \pm 0.30 & 132.94 \pm 2.11 & +1.69\\,\text{ms}\\,(+1.29\\%) \\\\ \text{CPU input to returned trajectory (ms)} & 134.48 \pm 0.19 & 135.01 \pm 0.07 & +0.53\\,\text{ms}\\,(+0.39\\%) \\\\ \hline \text{EPDMS (NAVSIM-v2 navhard)}\uparrow & 48.3 & \mathbf{54.9} & +6.6\\,\text{pts}\\,(+13.7\\%) \end{array}$$
Reviewer_Smm9OpenReview ↗

Response to authors

Thanks for the additional experiments and clarifications. They address most of my concerns. The static-target comparison helps, but as you noted, the targets come from different teacher layers. Could you compare static and future targets from the same layer? This would help clarify whether the extra gain comes from future dynamics.

Reviewer_9Mh3OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper adds a transition module to DrivoR. The module is supervised with features extracted offline from a frozen Wan2.2-TI2V-5B and modulates the current scene tokens through FiLM. A second loss distills features from the scorer into a set of physics tokens. The experiments show that the design is feasible to some extent.

Critical strengths

The idea of ​​using a large video model during training without paying for it at deployment is practical, and it requires only small changes to the base planner. The comparison in the appendix with directly replacing or concatenating future features is informative. The evaluation covers open-loop, pseudo-closed-loop, and zero-shot closed-loop settings.

Critical weaknesses
  1. The added module changes the architecture and capacity, so the current ablations do not establish how much of the gain comes from the teacher’s predictive knowledge.
  2. Section 4.3 states that the ablation does not establish motion disentanglement, yet the introduction and conclusion still describe the factors as disentangled from ego motion. Also, γ and β are unconstrained, so S′ can rewrite the original features entirely (γ = −1). The claim that imperfect dynamics can adjust but not replace the observed features therefore does not seem to hold.
  3. The method is tested with only one base planner (DrivoR) and one teacher (Wan2.2). How the module performs on other architectures, such as BEV-based TransFuser or diffusion-based DiffusionDrive, and whether it depends on DrivoR’s register-token design.
  4. The paper repeatedly says the inference overhead is negligible, but it reports no measured latency, parameter Count, or memory usage.
Questions for authors

How does the same architecture perform with λd = 0? What happens if static features are distilled instead?

Does the improvement hold when the module is applied to other planners, or when a different teacher model is used?

How the cached teacher features are generated: which features are taken from Wan, what generation settings are used, and how the teacher’s temporal features are aligned with the student’s prediction steps.

DriveFuture reports 55.5 EPDMS on navhard, compared with 54.9 here, but it is not included in Table 1. Given that Table 2 references the work on DriveFuture, could the authors clarify whether these two results are directly comparable?

4 comments
AuthorsOpenReview ↗

response-part1

We thank the reviewer for the careful and constructive review, and for recognizing the practicality of using a large video model only at training time. We respond to each point below, and the new results have been incorporated into the revised manuscript.

W1 & Q1 → A1

Thank you for raising this point. To separate the contribution of Wan2.2 supervision from that of the added architecture, we ran architecture-matched controls on NAVSIM-v2 navhard. All variants retain ego-conditioned pooling, both token branches, and FiLM modulation, and the training budget is held fixed.

Removing Wan2.2 supervision. The full model achieves 54.9 EPDMS. Setting λd=0\lambda_d=0, i.e., removing WAM distillation while keeping the architecture unchanged, reduces EPDMS to 50.9, a 4.0-point drop at fixed planner capacity. At least 4.0 of the 6.6-point gain over the DrivoR baseline (48.3) therefore depends on the Wan2.2 supervision.

Static targets. In the table below, we implement additional experiments replacing the future targets with static targets. This reaches 53.8, 2.9 above the variant without Wan2.2 supervision. Future-feature targets add a further 1.1, which demonstrates the feasibility of our method.

\text{WAM loss} & \text{Scorer distillation} & \lambda_d & \lambda_p & \text{EPDMS}\uparrow \\\\ \hline \times & \checkmark & 0 & 0.5 & 50.9 \\\\ \checkmark & \times & 1.0 & 0 & 52.4 \\\\ \times & \times & 0 & 0 & 49.2 \\\\ \hline \checkmark & \checkmark & 0.5 & 0.5 & 51.0 \\\\ \checkmark & \checkmark & 1.5 & 0.5 & 53.4 \\\\ \checkmark & \checkmark & 2.0 & 0.5 & 53.2 \\\\ \checkmark & \checkmark & 1.0 & 0.25 & 54.3 \\\\ \checkmark & \checkmark & 1.0 & 1.0 & 53.7 \\\\ \hline \text{Static Wan2.2 targets} & \checkmark & 1.0 & 0.5 & 53.8 \\\\ \checkmark & \checkmark & 1.0 & 0.5 & \mathbf{54.9} \end{array}$$ ### W2 → A2 Thank you for highlighting this issue. We agree that Eq. (2) is unconstrained and permits $\gamma=-1$, so it does not guarantee that the observed features are preserved. We examined the learned FiLM coefficients across 5,912 evaluated scenes. The mean and standard deviation were 0.698 and 1.803 for $\gamma$, and 0.057 and 0.733 for $\beta$. Fewer than 0.66% of token-channel scale coefficients $1+\gamma$ were within 0.01 of zero, based on an upper bound from the histogram, so near-cancellation of the direct observation term was uncommon. We have revised manuscript to describe the modulation of current scene features without claiming that it prevents replacement. We have also removed the claim that the factors are disentangled from ego motion in the Introduction and Conclusion. Ego conditioning improves navhard EPDMS from 52.3 to 54.4 (Table 4a), and the text now presents this as a measured planning benefit rather than evidence of disentanglement. ### W3 & Q2 → A3 Thank you for raising this point. We ran two additional experiments on NAVSIM-v2 navhard, changing the world-model teacher in one and the base planner in the other. **Different teacher.** Keeping the DrivoR planner and operator fixed, we replace the cached Wan2.2 targets with future features from frozen Cosmos-Predict2.5. The result, 54.7 EPDMS, is comparable to Wan2.2 (54.9) and 6.4 points above the DrivoR baseline, so the gain is not specific to Wan2.2. **Different planner.** We reproduced GTRS-Dense with its V2-99 backbone (44.2 EPDMS) and applied the ego-conditioned transition operator to the spatial feature grid of its image encoder, keeping its trajectory vocabulary and scoring pipeline and training the operator with cached Wan2.2 features. EPDMS rises to 46.5. The gain is smaller than on DrivoR, but it shows that the operator does not depend on DrivoR's register tokens. $$\begin{array}{llr} \text{Base planner} & \text{Configuration} & \text{EPDMS}\uparrow \\\\ \hline \text{DrivoR} & \text{Baseline} & 48.3 \\\\ \text{DrivoR} & \text{Ours + Wan2.2} & \mathbf{54.9} \\\\ \text{DrivoR} & \text{Ours + Cosmos-Predict2.5} & 54.7 \\\\ \hline \text{GTRS-Dense} & \text{Published reference} & 45.0 \\\\ \text{GTRS-Dense} & \text{Reproduced baseline} & 44.2 \\\\ \text{GTRS-Dense} & \text{Ours + Wan2.2} & \mathbf{46.5} \end{array}$$
AuthorsOpenReview ↗

response-part2

W4 → A4

Thank you for suggesting this comparison. We benchmarked DrivoR and our method in five paired runs on the same NVIDIA H20 GPU with a batch size of one and automatic mixed precision disabled. Model forward latency was 131.25 ± 0.30 ms for DrivoR and 132.94 ± 2.11 ms for our method, reported as the mean ± sample standard deviation across the five run means. The average overhead was 1.69 ms (1.29%). These measurements use synthetic inputs and exclude image preprocessing.

Estimated computation increased by 0.114 GFLOPs (approximately 0.0062%), from 1856.154 to 1856.268 GFLOPs. Inference parameter count and peak allocated GPU memory increased by 6.32% and 1.94%, respectively. We report the full comparison in Table 7 of Appendix C in the revised manuscript.

\text{Metric} & \text{DrivoR} & \text{Ours} & \text{Increase} \\\\ \hline \text{Inference parameters (M)} & 40.788 & 43.365 & +6.32\\% \\\\ \text{Model forward latency (ms)} & 131.25 \pm 0.30 & 132.94 \pm 2.11 & +1.29\\% \\\\ \text{Peak allocated GPU memory (MiB)} & 506.239 & 516.072 & +1.94\\% \\\\ \text{Estimated computation (GFLOPs)} & 1856.154 & 1856.268 & +0.0062\\% \\\\ \hline \text{EPDMS (NAVSIM-v2 navhard)} & 48.3 & \mathbf{54.9} & +13.7\\% \end{array}$$ ### Q3 → A5 For each training scene, we condition the frozen Wan2.2-TI2V-5B on the current front-camera image, resized to $320 \times 160$. We use an empty text prompt and disable classifier-free guidance (guidance scale 1.0). We sample a 33-frame latent sequence with 16 Flow-UniPC denoising steps and a sampling shift of 5.0, keeping the first latent slice anchored to the input image. We extract the output of the 16th DiT block (zero-based index 15 of 30) during the final denoising step. Wan2.2's temporal stride of four gives nine latent slices. We drop the anchored first slice and average-pool the spatial tokens within each of the eight remaining slices, producing one 3,072-dimensional feature per future step. We cache the resulting $8 \times 3072$ tensor offline by scene ID. The student's temporal head predicts eight feature vectors and matches the $k$-th prediction to the $k$-th cached vector using cosine loss. This alignment follows the teacher's latent time indices and is independent of the trajectory decoder's waypoint timestamps. We have added a brief description of the Wan2.2 feature extraction, offline caching, and temporal alignment to Appendix A of the revised manuscript. ### Q4 → A6 We thank the reviewer for raising this point. We have trained our model again, and the new model reaches 55.7 EPDMS on NAVSIM-v2 navhard, exceeding DriveFuture's reported 55.5. The table below compares the two methods on NAVSIM-v2 navhard and NAVSIM-v1 navtest. | Benchmark | Metric | DriveFuture | Ours | | --- | --- | --- | --- | | NAVSIM-v2 navhard | **EPDMS** | 55.5 | **55.7** | | NAVSIM-v1 navtest | **PDMS** | 90.7 | **93.8** | | Setting | Future-frame GT used in training | Yes | No | | | External GTRS-Dense scorer | Yes | No | | | World-model training | Joint with planner | None (frozen, cached features) | Compared with DriveFuture, our method uses no future frames, no external scorer, and no WAM training: the frozen WAM only provides cached features offline and is removed at inference. Our goal is therefore not to maximize the leaderboard score but to show how much a pretrained WAM can contribute under this lightweight setting. We will also release the latest model weights as open source.

Thank you for the additional experiments. The architecture-matched controls, second teacher and planner, and runtime measurements address most of my concerns. These clarifications address my main questions, and I look forward to seeing the modifications incorporated into the revised manuscript. I am raising my score to 3.

AuthorsOpenReview ↗

Thanks for raising score.

Thank you for raising your score. We agree that these points belong in the paper itself, and the revised version will state them explicitly.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Overall Response

We sincerely thank all reviewers and chairs for their time and effort in evaluating our paper. Below, we summarize the contributions recognized by the reviewers and our main updates during the rebuttal.

Contributions recognized by reviewers

A practical way to use large world models without paying for them at deployment. The frozen WAM serves only as a training-time teacher; a compact 16-factor transition operator modulates the current scene features through FiLM, and the WAM and all prediction heads are removed at inference. Training never uses a future frame, and hallucinated futures never reach the control path. The paper describes this design as "state-anchored." [9Mh3, Smm9, Rogk, cMYz]

Substantial gains across evaluation regimes. The method improves navhard EPDMS from 48.3 to 54.9 (+13.7%) and zero-shot HUGSIM HD-Score from 27.9 to 35.0 (+25.4%) over the base planner, without any rendered-frame training. Comparisons with direct future-feature replacement (51.6) and concatenation (51.1) support keeping WAM knowledge in a transition-based modulation of observed features (Table 5). [all reviewers]

A validated mechanism, not just a stronger planner. Architecture-matched ablations, teacher and planner transfer, and measured inference overhead show that the gains come from the distilled supervision at fixed planner capacity, and that the operator generalizes beyond the specific teacher (Wan2.2) and base planner (DrivoR) used in the main experiments. [9Mh3, Smm9, Rogk]

Additional experiments and clarifications

Attribution at fixed architecture. Reviewers asked whether the gains reflect transferred dynamics or added capacity. Retaining ego-conditioned pooling, both token branches, and FiLM modulation, removing WAM supervision reduces EPDMS from 54.9 to 50.9, and removing scorer supervision reduces it to 52.4. Removing both yields 49.2, only 0.9 above vanilla DrivoR. The distillations therefore contribute +5.7 points at constant capacity (Table 6). [9Mh3, Smm9, Rogk]

Transfer across teachers and planners. Replacing Wan2.2 with Cosmos-Predict2.5 gives 54.7 EPDMS. Applying the operator, without modification, to the spatial feature grid of GTRS-Dense improves our reproduced baseline from 44.2 to 46.5 under the same protocol, with no GTRS-specific tuning (Table 8). [9Mh3, Rogk]

Inference overhead, measured. Across five paired runs on an NVIDIA H20, forward latency increases by 1.29% (131.25 to 132.94 ms), with +6.32% inference parameters, +1.94% peak GPU memory, and +0.0062% estimated computation, alongside the +13.7% EPDMS gain (Table 7). We now report these numbers instead of describing the overhead qualitatively. [9Mh3, Smm9, Rogk]

Modulation and ego conditioning. We agree that unconstrained scale and shift coefficients, in principle, allow the transition term to suppress the observation term, and that our earlier evidence does not establish ego-motion disentanglement. Ego conditioning is now described consistently as supplying motion-state information during factor extraction, and the Introduction, Method, and Conclusion no longer claim disentanglement. [9Mh3, Smm9, cMYz]

Changes to the manuscript

  • Appendix A (Future-Feature Consumption Controls): added the teacher extraction and caching recipe (16th DiT block, final step of a 16-step denoising run, 33-frame latent with the image-anchored first slice discarded, spatial average pooling to an 8×3072 cache, cosine matching in temporal order), the two direct-future controls, and the note that these controls differ in downstream token counts.
  • Appendix B (Distillation Loss Ablations, Table 6): added architecture-matched controls for WAM supervision and scorer supervision, with several weight settings, keeping the full planner fixed in every row.
  • Appendix C (Inference Overhead, Table 7): added measured latency with per-run variation, parameter counts, memory usage, and computation estimates.
  • Appendix D (Transfer Across Teachers and Planners, Table 8): added the Cosmos-Predict2.5 teacher and the GTRS-Dense results.
  • Sections 1, 3, 4, and 5: revised the wording on ego conditioning, modulation behavior, and the scope of the closed-loop generalization claim. Table 1 now defines all nine EPDMS sub-metrics explicitly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.