acceptodds
Under review as a conference paper at ICLR 2027

DualWM: Dual-Layer Unified Asynchronous Latent World Model for End-to-End Autonomous Driving

Abstract

Latent world models enhance autonomous driving by learning task-state transitions, providing valuable policy priors. However, existing methods suffer from the fundamental conflict between cognitive reasoning and real-time responsiveness in dynamic driving scenarios. To address this challenge, we propose a novel end-to-end autonomous driving framework based on a Dual-layer Unified Asynchronous Latent World Model (DualWM) that learns state transitions at both the intent and behavioral levels within a unified latent space for efficient driving. Specifically, the VLM-based upper layer handles scene understanding and long-horizon intent, while the lower layer enables real-time semantic state prediction. The two layers are mutually coupled: the intent features provide long-horizon priors that guide behavioral prediction, and the behavioral states in turn constrain the uncertainty inherent in VLM predictions, while the asynchronous architecture improves inference efficiency. This design draws inspiration from human driving behavior, where conscious planning coexists with rapid reactive adjustments. Finally, a Mixture-of-Experts (MoE)-based policy module is designed to capture both the consistency and conflicts between macroscopic intent and behavioral-level decisions, enabling efficient and robust closed-loop driving. Extensive experiments on the Bench2Drive closed-loop benchmark in CARLA demonstrate that DualWM achieves strong closed-loop driving performance while substantially improving runtime efficiency, validating its potential for real-world autonomous driving.

Primary area:
applications to robotics, autonomy, planning
Keywords:
End-to-End Autonomous Driving, Latent World Models, Vision-Language Models, Asynchronous Inference, Mixture of Experts

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_tnmtOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents DualWM, an end-to-end driving framework coupling slowly updated VLM-derived intent with fast behavioral latent dynamics. Bidirectional feedback connects the two layers, while an intent-gated residual MoE planner generates trajectories. Experiments on Bench2Drive show strong closed-loop performance and substantially reduced average inference latency.

Critical strengths
  1. The separation between persistent maneuver intent and rapidly evolving behavioral dynamics is well motivated. The probabilistic formulation provides a coherent account of their interaction, supported by detailed implementation descriptions.

  2. DualWM achieves strong results of 81.22 DS and 61.51% SR, while reducing average latency from ORION’s 1493 ms to 551 ms on the same hardware. The component ablations show meaningful gains from feedback, predictive modeling, and expert planning.

Critical weaknesses
  1. The ablations do not fully isolate the benefit of the dual-layer dynamics. In Table 4, removing cognition also removes intent gating, while removing prediction affects both layers. A comparison that keeps the behavioral model and planner unchanged but replaces cognitive dynamics with cached VLM features would help clarify the specific contribution.

  2. The evidence for correcting VLM hallucinations is still limited. Figure 4 shows that the planner can compensate for an infeasible intent, but does not establish whether behavioral feedback improves the intent itself. The feedback ablation supports its overall usefulness; a direct comparison of intent accuracy or correction rates with and without feedback would better support the stronger claim.

  3. The reported average latency does not fully characterize the system’s responsiveness. Although Appendix C.4 explains how inference cost is measured, it remains unclear whether behavioral control continues during VLM inference and how much latency increases on query steps. Clarifying the execution schedule and reporting query-step versus nonquery-step latency would help assess the practical benefit of the asynchronous design.

Questions for authors
  1. What evidence distinguishes the benefit of learned cognitive dynamics from persistent VLM features, with the behavioral model and planner held fixed?

  2. Can the existing rollouts quantify whether feedback corrects erroneous decoded intents, rather than only compensating for them in the final trajectory?

  3. Does behavioral control continue while the VLM is running? Please clarify the execution schedule and provide query-step versus nonquery-step latency.

6 comments

We sincerely thank you for your careful, technically detailed review and constructive suggestions. We also appreciate your recognition of the motivation for separating persistent maneuver intent from rapidly evolving behavioral dynamics, our probabilistic formulation, and our closed-loop results. Over the approximately six weeks since submission, we have continued to improve our work and completed several additional experiments. We were particularly encouraged to find that the improvements we had been pursuing closely aligned with several of your suggestions. Your suggestions also helped us examine questions we had not previously investigated systematically and further strengthen the revised manuscript. Below, we address your comments point by point, supported by the relevant experimental results. Tables in this response are numbered independently as R1–R6. All manuscript section, table, and line references refer to the revised manuscript.

Critical Weaknesses

W1: The ablations do not fully isolate the benefit of the dual-layer dynamics. In Table 4, removing cognition also removes intent gating, while removing prediction affects both layers. A comparison that keeps the behavioral model and planner unchanged but replaces cognitive dynamics with cached VLM features would help clarify the specific contribution.

A1: Thank you for your detailed analysis and this targeted suggestion for a controlled comparison, which helps us examine the contribution of cognitive dynamics modeling more directly. The original cognitive-layer ablation also removed intent gating, while the prediction ablation disabled predictive modeling in both layers. These comparisons therefore assessed broader component contributions.

To examine the role of cognitive dynamics more directly, we refined the w/o cognitive dynamics control by replacing learned cognitive dynamics with cached VLM features while retaining the behavioral model, intent gating, and MoE planner. This control achieves 78.57 DS and 56.82% SR, compared with 81.44 DS and 60.45% SR for the full DualWM model. With the behavioral model and planner retained, the full model improves DS by 2.87 points and SR by 3.63 percentage points. These results support the additional benefit of learned cognitive dynamics beyond reusing cached VLM features. The results are shown below.

Table R1. Ablation on Bench2Drive. ΔDS is the DS change from the full model.

VariantDS ↑ΔDSSR (%) ↑Ability (%) ↑
DualWM (full)81.44—60.4562.31
w/o cognitive dynamics78.57−2.8756.8255.60

We have revised the manuscript and added these results and the control settings to Table 5 in the main text (lines 433–445) and Appendix D.3 (lines 1428–1440).

W2: The evidence for correcting VLM hallucinations is still limited. Figure 4 shows that the planner can compensate for an infeasible intent, but does not establish whether behavioral feedback improves the intent itself. The feedback ablation supports its overall usefulness; a direct comparison of intent accuracy or correction rates with and without feedback would better support the stronger claim.

A2: Thank you for your suggestion, which helped us present the effects of feedback on intent prediction and behavioral compensation more clearly in the paper.

① In our architecture, VLM hallucinations primarily manifest as incorrect cognitive intents.

To assess whether behavioral feedback improves intent prediction itself, we compare DualWM with a variant that removes bottom-up behavioral-state feedback while retaining top-down cognitive conditioning. As shown in Table R2, adding feedback improves intent accuracy from 74.80% to 83.98% when both variants are evaluated on the same validation segments. This indicates that our lower-level behavioral dynamics representation provides useful feedback to the higher-level cognitive intent representation, helping reduce intent prediction errors.

② We also evaluate the behavioral layer's ability to compensate for incorrect cognitive guidance. Compensation denotes the proportion of cases in which the behavioral intent derived from the predicted trajectory is correct, given that the cognitive intent is incorrect. Adding feedback increases this proportion from 69.23% to 73.08%, indicating that DualWM has a stronger ability to compensate for incorrect cognitive guidance at the behavioral level.

③ Finally, to further assess the effect of behavioral feedback on closed-loop driving safety, we measure the proportions of routes with collisions and traffic violations. Adding feedback reduces the proportion of routes with collisions from 35.00% to 30.15% and the proportion with traffic violations from 20.00% to 14.85%, further supporting the contribution of behavioral feedback to safer driving. The closed-loop safety analysis is provided in Appendix D.3 (lines 1483–1487). The results are shown below.

Table R2. Effect of bottom-up feedback on intent prediction and driving performance.

VariantIntent accuracy ↑Compensation ↑Collision ↓Violation ↓
w/o feedback74.8069.2335.0020.00
DualWM83.9873.0830.1514.85

We have revised the manuscript and added the feedback experiment results to Table 13 in Appendix D.3 (lines 1513–1521).

W3: The reported average latency does not fully characterize the system’s responsiveness. Although Appendix C.4 explains how inference cost is measured, it remains unclear whether behavioral control continues during VLM inference and how much latency increases on query steps. Clarifying the execution schedule and reporting query-step versus nonquery-step latency would help assess the practical benefit of the asynchronous design.

A3: Thank you for the opportunity to clarify this point, which also helps us explain more clearly how we evaluate the practical benefits of our asynchronous design.

To clarify how behavioral control operates during VLM inference, we supplemented the standard synchronous evaluation with a latency-injection experiment. Standard synchronous evaluation assesses the method's performance under idealized timing conditions without inference-delay effects, whereas latency injection incorporates the measured inference time into simulation time, more closely reflecting real-world driving conditions. We also introduce an asynchronous execution mechanism to mitigate the effects of long inference delays during policy deployment. Below, we first describe the execution schedule in the standard synchronous evaluation, followed by DualWM's asynchronous execution mechanism and its latency-injection evaluation results.

A3 (continued):

① In the standard synchronous Bench2Drive/CARLA evaluation used for DualWM and the baselines, behavioral control does not continue while the VLM is running. Instead, the next control command is generated only after VLM inference finishes. CARLA also waits for the model output before advancing the simulation. Thus, the simulated environment does not advance during this wait, and the wall-clock inference time does not affect driving performance in this setting.

On the A800, query steps have a mean latency of 1401.16 ms and a p95 of 1477.97 ms, while non-query steps have a mean latency of 167.23 ms and a p95 of 176.06 ms. The latency distributions on both GPUs are shown below.

Table R3. Latency statistics of DualWM on the A800 and H20 at K=10 (ms).

GPUScopeMeanMedianP95Max
A800Query step1401.161389.581477.972230.83
A800Non-query step167.23167.28176.06232.99
A800All steps291.66167.991390.282230.83
H20Query step1077.281053.611128.021767.83
H20Non-query step164.26164.17167.43182.83
H20All steps257.07164.631053.831767.83

We have revised the manuscript and added these latency statistics to Table 9 in Appendix D.1 (lines 1363–1380).

These results show that query-step latency is substantially higher than non-query-step latency. In real-world driving, however, the environment continues to evolve during inference, and long tail latencies can block control updates, limiting real-time deployment.

② To reduce the impact of long tail latencies on real-time deployment of closed-loop driving policies, our method uses DualWM's asynchronous dual-layer architecture during inference, allowing behavioral control to continue while the VLM is running. Until the VLM returns the intent features for the current query, the previous intent features remain in use, while the behavioral layer can continue adjusting the driving plan as it awaits the cognitive update. Because the intent features provide high-level guidance that remains relatively stable over a longer time window, they can still effectively support current action generation.

Under this asynchronous execution scheme, VLM queries run in parallel with the behavioral layer, and their cognitive outputs can be reused across multiple control steps, allowing behavioral control to continue while a cognitive update is pending. In the latency-injection evaluation, control-generation times remain broadly comparable across steps, averaging 182.41 ms per step, close to the 167.23 ms measured for non-query steps in the synchronous evaluation. This reduces the blocking effects of query-step tail latency on control updates. The latency distribution is shown in Table R4 below.

Table R4. Control-generation latency in the latency-injection evaluation (ms).

GPUScopeMeanMedianP95Max
A800All steps182.41174.50232.70349.09

We have revised the manuscript and added the control-generation latency results and analysis to Appendix D.2, with the results reported in Table 10 (lines 1420–1424).

③ To further evaluate driving performance under inference delays with this asynchronous execution mechanism, we added a latency-injection experiment using measured A800 delays. In this setting, the simulated world continues advancing while outputs are pending, and the vehicle holds its previous control command until a new one becomes available, to simulate real-world driving conditions.

Under the same latency-injection setting, DualWM achieves 71.06 DS, compared with 35.36 for ORION, 35.90 for MindDrive, and 18.40 for DriveMoE. Relative to their respective scores without latency injection, ORION, MindDrive, and DriveMoE show DS reductions of 54.52%, 54.00%, and 75.21%, respectively. DualWM retains 87.25% of its original DS, a reduction of 12.75%. These results show that DualWM maintains better closed-loop driving performance than the other tested methods when inference delays are taken into account. The results are shown below.

Table R5. Closed-loop results under measured inference latency injection on Bench2Drive.

MethodDS ↑SR (%) ↑RC (%) ↑Mean abilities (%) ↑
DriveMoE18.400.4558.613.41
ORION35.369.0972.5518.05
MindDrive35.908.6470.4616.74
DualWM71.0642.2793.4347.42

We have revised the manuscript and added these results and analyses to Table 4 in the main text (lines 399–414) and Appendix D.2 (lines 1383–1395).

Reviewer_xkMjOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DualWM couples a VLM-based cognitive intent state updated every ten frames with a per-frame BEV behavioral state. Intent conditions behavioral modeling, while behavioral history and executed actions inform subsequent cognitive updates. Four intent-gated residual experts refine a base trajectory. Future prediction primarily supports representation learning during training; deployment generates trajectories directly from the latent states. On 220 Bench2Drive routes with three traffic seeds, the paper reports 81.22 DS, 61.51% SR, and a mean end-to-end latency of 551 ms per step.

Critical strengths
  1. The bidirectional coupling is well motivated and has preliminary empirical support. Conditioning cognitive updates on behavioral history and executed actions provides a plausible way to keep low-frequency intent consistent with the driving state. Removing behavioral feedback reduces DS by 5.17 points (Table 4), supporting the usefulness of this pathway.

  2. The trade-off between closed-loop performance and computation is valuable. DualWM exceeds ORION's published results by 3.48 DS and 6.89 percentage points in SR, while mean inference latency measured on the same hardware falls from 1493 ms to 551 ms. Evaluation over the full route set and per-traffic-seed results strengthen the evidence for the overall gains.

Critical weaknesses
  1. The distinct benefits of dual-layer modeling and feedback-based intent correction are not sufficiently isolated. “w/o cognition” removes both cognitive conditioning and intent gating. “w/o prediction” retains temporal encoding and MoE but disables prediction in both layers (Appendix D.1). These results support prediction and feedback, but do not fully establish the additional value of two latent processes over a simpler recurrent model with comparable supervision. Figure 4(b) also demonstrates behavioral compensation for an incorrect intent, without establishing that feedback corrects the intent representation itself.

  2. Computational savings do not establish timely responses under actual inference delays. Appendix C.4 explicitly distinguishes the 10 Hz simulation rate from the 551 ms mean wall-clock latency. However, Algorithm 2 leaves VLM-query blocking behavior unspecified, and query-step latency distributions and the treatment of observation-to-action delay remain unreported. Emergency-braking success is 66.67%, versus ORION's 78.33% (Table 2). This warrants further explanation, although the difference cannot be attributed to inference latency from the available evidence.

Questions for authors
  1. Do VLM queries block control? Please report mean and P95 latency separately for query and non-query steps, and explain how closed-loop evaluation handles actual observation-to-action delay.

  2. Can you provide a single-layer recurrent baseline using cached VLM features, or an equivalent controlled ablation, with matched encoders, supervision, MoE, query frequency, and approximately matched parameter count, to assess the additional value of the dual-layer structure?

  3. Are the descriptions in Figure 4 raw model outputs, decoded intent labels, or human annotations? For matched cases with initially incorrect intents, how do intent changes and collision/infraction outcomes differ with and without feedback? This would distinguish intent correction from trajectory-level compensation.

8 comments

We sincerely thank you for your careful, technically detailed review and constructive suggestions. We also appreciate your recognition of the motivation behind our bidirectional coupling and the trade-off between closed-loop performance and computational cost. Over the approximately six weeks since submission, we have continued to improve our work and completed several additional experiments. We were particularly encouraged to find that the improvements we had been pursuing closely aligned with several of your suggestions. Your suggestions also helped us examine questions we had not previously investigated systematically and further strengthen the revised manuscript. Below, we address your comments point by point, supported by the relevant experimental results. Tables in this response are numbered independently as R1–R7. All manuscript section, table, figure, and line references refer to the revised manuscript.

Critical Weaknesses

W1: The distinct benefits of dual-layer modeling and feedback-based intent correction are not sufficiently isolated. “w/o cognition” removes both cognitive conditioning and intent gating. “w/o prediction” retains temporal encoding and MoE but disables prediction in both layers (Appendix D.1). These results support prediction and feedback, ① but do not fully establish the additional value of two latent processes over a simpler recurrent model with comparable supervision. ② Figure 4(b) also demonstrates behavioral compensation for an incorrect intent, without establishing that feedback corrects the intent representation itself.

A1: Thank you for pointing out what our original ablations did not clearly distinguish. This helped us focus more closely on the respective roles of the dual-layer structure and feedback.

① We expanded our ablation studies by evaluating a single-layer recurrent baseline that replaces the two latent processes with a deterministic GRU, while retaining the same encoders, MoE planner, query interval, supervision, and training settings, with comparable model capacity. This baseline achieves 75.81 DS, compared with 81.44 for DualWM. The 5.63-point gap supports the additional value of modeling cognition and behavior as two coupled latent processes over a simpler recurrent model.

Your suggestion also prompted us to examine the role of cognitive dynamics modeling. We therefore conducted an additional evaluation of the “w/o cognitive dynamics” variant, which retains the behavioral world model but replaces learned cognitive dynamics with cached VLM features. It achieves 78.57 DS, which is still 2.87 points below the full model. Together, these results support the value of the dual-layer structure over both a single-layer recurrent model and a behavioral world model conditioned on cached VLM features. The results are shown below.

Table R1. Ablation on Bench2Drive. ΔDS is the DS change from the full model.

VariantDS ↑ΔDSSR (%) ↑Ability (%) ↑
DualWM (full)81.44—60.4562.31
w/o cognitive layer67.13−14.3140.0035.86
w/o cognitive dynamics78.57−2.8756.8255.60
w/o behavioral layer76.42−5.0250.9153.80
w/o dual layer (GRU)75.81−5.6352.7352.49
w/o feedback77.14−4.3052.2752.92
w/o prediction71.15−10.2941.3640.99
w/o MoE73.88−7.5646.3643.84

② We also sincerely appreciate your suggestion to distinguish intent correction from behavioral compensation.

Figure 5(b) in the revised manuscript (originally Figure 4(b)) illustrates behavioral compensation for an incorrect intent. To further examine whether feedback also improves cognitive intent prediction, we added a controlled comparison with and without feedback to evaluate intent prediction accuracy. Specifically, we first compared DualWM with its no-feedback variant on the validation set. With feedback, intent accuracy increases from 74.80% to 83.98%. This result shows that bottom-up behavioral feedback helps improve high-level cognitive intent prediction.

This suggestion also prompted us to examine the effect of feedback on closed-loop driving safety. On the same 220 closed-loop routes, adding feedback reduces the proportion of routes with collisions from 35.00% to 30.15%, and the proportion with traffic violations from 20.00% to 14.85%. These results provide quantitative evidence that feedback improves both intent prediction itself and closed-loop driving safety. The results are shown below.

Table R2. Effect of bottom-up feedback on intent prediction and driving performance.

VariantIntent accuracy ↑Compensation ↑Collision ↓Violation ↓
w/o feedback74.8069.2335.0020.00
DualWM83.9873.0830.1514.85

We have updated Table 5 (lines 433–445) and Appendix D.3 (lines 1428–1440; Table 13, lines 1513–1521).

W2: ① Computational savings do not establish timely responses under actual inference delays. Appendix C.4 explicitly distinguishes the 10 Hz simulation rate from the 551 ms mean wall-clock latency. However, Algorithm 2 leaves VLM-query blocking behavior unspecified, and query-step latency distributions and the treatment of observation-to-action delay remain unreported.

A2: We fully understand your concern about control responsiveness under actual inference delays. This is an important issue.

A2① Algorithm 2 describes the standard synchronous Bench2Drive/CARLA evaluation, in which CARLA waits for each model update to finish, including any VLM query. Consequently, the wall-clock waiting time caused by VLM-query blocking does not affect driving performance in this simulation setting.

We also report latency distributions separately for query and non-query steps in this synchronous evaluation. Their mean latencies on the A800 are 1401.16 ms and 167.23 ms, respectively. The median, p95, and maximum are shown below.

Table R3. Latency statistics of DualWM on the A800 at K=10 (ms).

ScopeMeanMedianP95Max
Query step1401.161389.581477.972230.83
Non-query step167.23167.28176.06232.99
All steps291.66167.991390.282230.83

We have revised the manuscript and added these latency statistics to Table 9 in Appendix D.1 (lines 1363–1380).

In real driving, however, such long VLM inference delays on query steps can block control updates and hinder real-time deployment.

To mitigate this effect, our method uses DualWM's asynchronous dual-layer architecture at inference time. Until the VLM returns the intent features for the current query, the previous intent features remain in use. Meanwhile, the behavioral layer continues processing new observations and combines them with the latest available cognitive guidance to generate new trajectories. Because the intent features provide high-level guidance that remains relatively stable over a longer time window, they can still effectively support current action generation.

Under this asynchronous execution scheme, VLM queries run in parallel with the behavioral layer, and their cognitive outputs can be reused across multiple control steps. The behavioral layer can therefore continue generating control commands while a cognitive update is pending. In the latency-injection evaluation, control-generation times remain broadly comparable across steps, with a mean of 182.41 ms per step, close to the 167.23 ms for non-query steps in the synchronous evaluation. This reduces the blocking effects of query-step tail latency on control updates. The latency distribution is shown in Table R4.

Table R4. Control-generation latency in the latency-injection evaluation (ms).

GPUScopeMeanMedianP95Max
A800All steps182.41174.50232.70349.09

We have revised the manuscript and added these results and analyses to Appendix D.2, with the results reported in Table 10 (lines 1420–1424).

To address the demands of real-time driving and assess the impact of inference delays on closed-loop performance more directly, we added a latency-injection experiment using measured A800 delays. We compare the above execution scheme with other methods under these delays. In this experiment, the simulated world continues advancing while outputs are pending, and the vehicle holds the previous control command until a new one becomes available, to simulate real-world driving conditions.

Under the same latency-injection setting, DualWM achieves 71.06 DS, compared with 35.36 for ORION, 35.90 for MindDrive, and 18.40 for DriveMoE. Relative to their respective scores without latency injection, the DS scores of ORION, MindDrive, and DriveMoE decrease by 54.52%, 54.00%, and 75.21%, respectively. DualWM retains 87.25% of its original DS. These results show that DualWM maintains better closed-loop driving performance than the other tested methods when inference delays are taken into account. The experimental results are shown below.

Table R5. Closed-loop results under measured inference latency injection on Bench2Drive.

MethodDS ↑SR (%) ↑RC (%) ↑Mean abilities (%) ↑
DriveMoE18.400.4558.613.41
ORION35.369.0972.5518.05
MindDrive35.908.6470.4616.74
DualWM71.0642.2793.4347.42

We have revised the manuscript and added these results and analyses to Table 4 in the main text (lines 399–414) and Appendix D.2 (lines 1383–1395).

W2: ② Emergency-braking success is 66.67%, versus ORION's 78.33% (Table 2). This warrants further explanation, although the difference cannot be attributed to inference latency from the available evidence.

A2② Regarding emergency braking, we would like to clarify two points.

First, DualWM's emergency-braking success rate remains below ORION's by 8.33 percentage points in the revised results. This gap may reflect a trade-off between emergency braking and normal progress through traffic. If a method places too much emphasis on emergency braking alone, its driving policy may favor conservative slowing or stopping, while unnecessary braking can impede normal progress and reduce overall efficiency. Our goal is to balance timely braking with appropriate progress while maintaining performance across different driving abilities. The revised results show that DualWM outperforms ORION on most other metrics, with clear advantages in Give Way and Efficiency. Across these metrics, DualWM achieves better overall driving performance. The results are shown below.

Table R6. Closed-loop results on Bench2Drive (ORION and DualWM rows).

MethodVenueSchemeModalityDS ↑SR (%) ↑Efficiency ↑
ORIONICCV 25ILC77.7454.62151.48
DualWM—ILC81.4460.45185.15

Table R7. Multi-ability success rates (%) (ORION and DualWM rows).

MethodSchemeMerging ↑Overtaking ↑Emergency Brake ↑Give Way ↑Traffic Sign ↑Mean ↑
ORIONIL25.0071.1178.3330.0069.1554.72
DualWMIL47.0871.8570.0053.3369.3062.31

The relevant results are reported in Tables 1–2 in the main text (lines 325–343 and 379–396).

Second, ORION's official closed-loop implementation adopts a relatively conservative speed-control policy, with a threshold of 5 m/s (18 km/h). Once the vehicle exceeds this threshold, throttle input is restricted to limit further acceleration. Under comparable braking conditions, a lower approach speed can shorten the required stopping distance, so this more conservative speed policy may contribute to its emergency-braking performance. Our speed limit is 10 m/s (36 km/h), which more closely reflects practical driving requirements. This difference in speed-control policies may also partly explain why our emergency-braking success rate is slightly lower than ORION's.

Questions

Q1: Do VLM queries block control? Please report mean and P95 latency separately for query and non-query steps, and explain how closed-loop evaluation handles actual observation-to-action delay.

Reviewer_NgRgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes DualWM, an end-to-end driving framework built around a two-timescale latent world model. A slow "cognitive" layer consumes the planning token of a frozen VLM (ORION) every K=10 frames and maintains a recurrent intent state; a fast "behavioral" layer is an RSSM-style BEV dynamics model over frozen BEVFormer features, conditioned on the held intent state. The layers are coupled top-down (intent conditions the behavioral transition) and bottom-up (behavioral states from the last interval and the executed action chunk condition the next cognitive update). The model is trained with a two-timescale sequential ELBO plus a future-prediction term, and a planner adds intent-gated residual experts (brake/bypass/merge/turn) to a base trajectory. On Bench2Drive (base set, 220 routes) DualWM reports 81.22 DS / 61.51% SR, above ORION (77.74) and LVDrive (80.71), with an average per-step latency of 551 ms versus 1493 ms for ORION. Ablations remove cognition, bottom-up feedback, prediction, and the MoE head.

Critical strengths
  1. Strong closed-loop result with a sensible amortization idea. Reusing a frozen VLM at 1/10 the rate while a lightweight recurrent world model fills the gap is a practical, well-motivated design, and the reported DS/SR (Table 1) are the best among the listed base-set methods, with low across-seed variance (Table 8: DS 81.22 ± 0.44). The multi-ability gains on Merging and Give Way (Table 2) are consistent with the claim that structured intent plus behavior experts helps interactive maneuvers.
  2. Clean, explicit probabilistic formulation and documentation. The two-timescale factorization (Eqs. 1–2, 8–9) is stated precisely, the departures of the implemented loss from the bound are listed (App. A.7), and the inference schedule (Alg. 2) is unambiguous. This makes the system reproducible and easy to extend.
Critical weaknesses
  1. The efficiency claim is not established in the form it is stated. (a) The 551 ms figure is a mean over control steps. On query frames DualWM still runs the full ORION forward pass, so worst-case per-step latency is at least ORION's (~1.5 s) plus the world-model cost. For closed-loop control the tail latency, not the amortized mean, determines feasibility. Even the mean is 5.5× slower than the 100 ms control period, and Bench2Drive/CARLA runs synchronously, so latency has no effect on the reported driving scores. (b) In Table 3 only methods marked † were re-measured on the same hardware; ThinkTwice, DriveAdapter and LVDrive latencies come from elsewhere, so the cross-method comparison (e.g., "27.7% lower than ThinkTwice") is not apples-to-apples. (c) The non-VLM part of the pipeline costs roughly 400 ms per step (551 ms minus ~149 ms of amortized ORION). This needs a breakdown, since it bounds what asynchrony can buy. The paper should report per-step latency distributions (median and p95/max), query-frame versus non-query-frame cost, and all baselines on identical hardware, and should temper "toward real-time deployment".
  2. The contribution over the frozen ORION backbone is not isolated, and there is a selection concern. Removing cognition drops DS from 81.22 to 62.88 (Table 4), so most of the performance comes from ORION's planning token, which is already trained for planning on Bench2Drive. The key missing baselines are: (i) ORION queried every K frames with its trajectory or plan held, or naively interpolated, between queries; (ii) DualWM with K=1, which separates the effect of the world model and MoE head from the effect of asynchrony; (iii) the fast-slow designs the paper itself cites (Sec. 2.3) under the same backbone. Separately, App. D.2 states that K was chosen by a closed-loop sweep over the evaluation routes using an earlier checkpoint. That conflicts with the statement in App. C.4 that "no evaluation-route information" was used for hyperparameter selection. The earlier checkpoint scores 76.75 DS at K=10, versus 81.22 for the final one. This 4.5-point checkpoint-to-checkpoint gap is about 10× the reported seed std, which suggests the three "seeds" (App. D.3 describes them as traffic-manager seeds of one model) understate the real variance. Please clarify how the final checkpoint was chosen and whether the three seeds are training seeds or evaluation seeds.
Questions for authors
  1. Are the three seeds in Table 8 independent training runs, or one trained model evaluated with three traffic-manager seeds? How was the final checkpoint chosen (validation loss or closed-loop routes)?
  2. Please report latency for query frames vs. non-query frames separately, plus median/p95, with every Table 3 baseline measured on the same A800. What dominates the ~400 ms non-VLM cost?
  3. What are the results of (a) ORION every 10 frames with a held or interpolated plan, and (b) DualWM with K=1? Without these, it is unclear whether the gain over ORION comes from the world model and MoE head or from the asynchrony.
  4. The claim that the behavioral layer "constrains VLM hallucinations" rests on the qualitative Fig. 4 and the w/o-feedback ablation. Can you measure it directly? For example: intent-head accuracy on the validation clips with and without bottom-up feedback, or the rate at which a wrong ORION intent is corrected. Note that with λ_KL = 1e-3 the cognitive KL term (the stated mechanism) is weakly weighted.

Comments on the paper

9 comments

We sincerely thank you for your careful, technically detailed review and constructive suggestions. We also appreciate your recognition of our method’s practical motivation, closed-loop performance, and clear probabilistic formulation. Over the approximately six weeks since submission, we have continued to improve our work and completed several additional experiments. We were particularly encouraged to find that the improvements we had been pursuing closely aligned with several of your suggestions. Many of your suggestions also helped us examine questions we had not previously investigated systematically and further strengthen the revised manuscript. Below, we address your comments point by point, supported by the relevant experimental results. Tables in this response are numbered independently as R1–R7. All manuscript section, table, and line references in our responses refer to the revised manuscript.

Critical Weaknesses

W1 — The efficiency claim is not established in the form it is stated.

W1(a): The 551 ms figure is a mean over control steps. On query frames DualWM still runs the full ORION forward pass, so worst-case per-step latency is at least ORION's (~1.5 s) plus the world-model cost. For closed-loop control the tail latency, not the amortized mean, determines feasibility. Even the mean is 5.5× slower than the 100 ms control period, and Bench2Drive/CARLA runs synchronously, so latency has no effect on the reported driving scores.

A1(a): Thank you for your detailed summary. We fully understand your concern about how inference latency affects closed-loop control. This is a key issue.

① First, after profiling the time spent at each stage, we made targeted optimizations to the most time-consuming steps. We are pleased to share the updated inference latency results. After optimization, the mean per-step latency on the A800 has decreased from 551 ms to 291.66 ms. Query steps average 1401.16 ms, while non-query steps average 167.23 ms. To address your concern about the worst case, we also report the median, p95, and maximum observed latency. The maximum observed query-step latency is 2230.83 ms. The A800 latency distributions are shown below.

Table R1. Latency statistics of DualWM on the A800 at K=10 (ms).

ScopeMeanMedianP95Max
Query step1401.161389.581477.972230.83
Non-query step167.23167.28176.06232.99
All steps291.66167.991390.282230.83

We have revised the manuscript and added the latency statistics to Table 9 in Appendix D.1 (lines 1363–1380).

As shown above, query-step latency is substantially higher than non-query-step latency. In real-world driving, however, the environment continues to evolve during inference, and long tail latency can block control updates and constrain real-time deployment.

② To reduce the impact of long tail latencies on real-time deployment of closed-loop driving policies, our method can use its asynchronous dual-layer architecture at inference time. Until the VLM returns the intent features for the current query, the previous intent features remain in use. Meanwhile, the behavioral layer continues processing new observations and combines them with the latest available cognitive guidance to generate new trajectories. Because the intent features provide high-level guidance that remains relatively stable over a longer time window, they can still support current action generation.

Under this asynchronous execution scheme, VLM queries run in parallel with the behavioral layer, and their cognitive outputs can be reused across multiple control steps. Although request handling and concurrent resource use introduce some additional overhead, the behavioral layer can continue generating control commands while a cognitive update is pending. In the latency-injection evaluation, control-generation times remain broadly comparable across steps, with a mean of 182.41 ms per step, close to the 167.23 ms for non-query steps in the synchronous evaluation. This reduces the blocking effects of query-step tail latency on control updates. The latency distribution is shown below.

Table R2. Control-generation latency in the latency-injection evaluation (ms).

GPUScopeMeanMedianP95Max
A800All steps182.41174.50232.70349.09

We have revised the manuscript and added the control-generation latency statistics to Table 10 in Appendix D.2 (lines 1420–1424).

③ In addition, standard Bench2Drive/CARLA evaluation runs synchronously, with the simulation advancing only after the model output is available. It therefore does not capture the effect of inference delays on closed-loop driving performance, as you correctly noted. Comparisons of mean inference latency can still provide an indirect indication of computational efficiency across methods.

To evaluate our method's driving performance under inference delays more directly, we have added a latency-injection experiment based on measured A800 delays. In this setting, the simulated world continues to advance while outputs are pending, and the vehicle maintains its previous control command until a new command becomes available, thereby simulating real-world driving conditions.

Under the same latency-injection setting, DualWM achieves 71.06 DS, compared with 35.36 for ORION, 35.90 for MindDrive, and 18.40 for DriveMoE. Relative to their respective scores without latency injection, ORION, MindDrive, and DriveMoE show DS reductions of 54.52%, 54.00%, and 75.21%, respectively. DualWM retains 87.25% of its original 81.44 DS. These results show that DualWM maintains better closed-loop driving performance than the other tested methods when inference delays are taken into account. The results are shown below.

Table R3. Closed-loop results under measured inference latency injection on Bench2Drive.

MethodDS ↑SR (%) ↑RC (%) ↑Mean abilities (%) ↑
DriveMoE18.400.4558.613.41
ORION35.369.0972.5518.05
MindDrive35.908.6470.4616.74
DualWM71.0642.2793.4347.42

We have revised the manuscript and added the latency-injection results to Table 4 in the main text (lines 399–414), with the accompanying analysis in lines 430–456.

W1(b): In Table 3 only methods marked † were re-measured on the same hardware; ThinkTwice, DriveAdapter and LVDrive latencies come from elsewhere, so the cross-method comparison (e.g., "27.7% lower than ThinkTwice") is not apples-to-apples.

A1(b): Thank you for your suggestion. For a fairer comparison of inference latency, we have remeasured ThinkTwice and DriveAdapter on the same A800 hardware used for DualWM. Since LVDrive's code was unavailable when we conducted the measurements, we use the H20 latency reported in its paper (2030 ms). We have also remeasured all the other methods on the H20.

On both GPU platforms, our method has lower mean inference latency than VLM-based driving methods such as MindDrive, ORION, and DriveMoE. Compared with MindDrive, the fastest of these baselines, latency is reduced by 73.24% on the A800 and 65.69% on the H20. The results are summarized below.

Table R4. Inference latency on Bench2Drive. † denotes our measurements on the respective GPU.

MethodLLMDS ↑A800 latency (ms) ↓H20 latency (ms) ↓
ThinkTwice†—62.44431394
DriveAdapter†—64.22443404
DriveMoE†PaliGemma-3B74.221116942
ORION†Vicuna-v1.5-7B77.7414931166
MindDrive†Qwen2-0.5B78.041091749
LVDriveVicuna-v1.5-7B80.71—2030
DualWM†Vicuna-v1.5-7B81.44292257

We have revised the manuscript and added these details to Table 3 (lines 399–414) and the accompanying analysis in the main text (lines 419–429).

W1(c): ① The non-VLM part of the pipeline costs roughly 400 ms per step (551 ms minus ~149 ms of amortized ORION). This needs a breakdown, since it bounds what asynchrony can buy. ② The paper should report per-step latency distributions (median and p95/max), query-frame versus non-query-frame cost, and all baselines on identical hardware, and should temper "toward real-time deployment".

A1(c): Thank you for this suggestion, which helped us improve our analysis of inference efficiency.

① In the synchronous Bench2Drive/CARLA evaluation, the non-VLM components in the original version averaged 440.44 ms per step in total, including 389.96 ms for BEV encoding. In the latest A800 measurements, the mean non-VLM cost is 174.63 ms per step, including 158.57 ms for BEV encoding and 8.86 ms for the world model and planner combined. BEV encoding therefore remains the main computational bottleneck outside the VLM. The breakdown is shown below.

Table R5. Mean latency breakdown before and after optimization (ms/step).

ComponentOriginal A800Optimized A800Optimized H20
Image preparation34.622.631.50
BEV encoding389.96158.57161.47
VLM overhead110.98117.0385.53
World model and planning9.528.865.96
Other (incl. PID)6.344.572.62
Total551.42291.66257.07

We have revised the manuscript and added the latency breakdown to Table 7 in Appendix D.1 (lines 1351–1361).

② For the standard synchronous Bench2Drive/CARLA evaluation, Table R1 in A1(a) reports the mean, median, p95, and maximum latency for query steps, non-query steps, and all steps on the A800. The H20 distributions have also been added to manuscript Table 9. For the baseline comparison on the same hardware, see Table R4 in A1(b) above.

For the latency-injection experiment, we additionally measured the latency distribution for generating each new control command under this asynchronous execution scheme. The reported mean control-generation latency is 182.41 ms, and the distribution is shown in Table R2 above. These statistics indicate that control-generation times remain broadly comparable across steps. By allowing control updates without waiting for a complete VLM query, this execution scheme reduces the blocking effects of query-step tail latency and improves control responsiveness, bringing DualWM closer to real-time deployment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.