- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper presents DualWM, an end-to-end driving framework coupling slowly updated VLM-derived intent with fast behavioral latent dynamics. Bidirectional feedback connects the two layers, while an intent-gated residual MoE planner generates trajectories. Experiments on Bench2Drive show strong closed-loop performance and substantially reduced average inference latency.
Critical strengths
-
The separation between persistent maneuver intent and rapidly evolving behavioral dynamics is well motivated. The probabilistic formulation provides a coherent account of their interaction, supported by detailed implementation descriptions.
-
DualWM achieves strong results of 81.22 DS and 61.51% SR, while reducing average latency from ORION’s 1493 ms to 551 ms on the same hardware. The component ablations show meaningful gains from feedback, predictive modeling, and expert planning.
Critical weaknesses
-
The ablations do not fully isolate the benefit of the dual-layer dynamics. In Table 4, removing cognition also removes intent gating, while removing prediction affects both layers. A comparison that keeps the behavioral model and planner unchanged but replaces cognitive dynamics with cached VLM features would help clarify the specific contribution.
-
The evidence for correcting VLM hallucinations is still limited. Figure 4 shows that the planner can compensate for an infeasible intent, but does not establish whether behavioral feedback improves the intent itself. The feedback ablation supports its overall usefulness; a direct comparison of intent accuracy or correction rates with and without feedback would better support the stronger claim.
-
The reported average latency does not fully characterize the system’s responsiveness. Although Appendix C.4 explains how inference cost is measured, it remains unclear whether behavioral control continues during VLM inference and how much latency increases on query steps. Clarifying the execution schedule and reporting query-step versus nonquery-step latency would help assess the practical benefit of the asynchronous design.
Questions for authors
-
What evidence distinguishes the benefit of learned cognitive dynamics from persistent VLM features, with the behavioral model and planner held fixed?
-
Can the existing rollouts quantify whether feedback corrects erroneous decoded intents, rather than only compensating for them in the final trajectory?
-
Does behavioral control continue while the VLM is running? Please clarify the execution schedule and provide query-step versus nonquery-step latency.
6 comments
W2: The evidence for correcting VLM hallucinations is still limited. Figure 4 shows that the planner can compensate for an infeasible intent, but does not establish whether behavioral feedback improves the intent itself. The feedback ablation supports its overall usefulness; a direct comparison of intent accuracy or correction rates with and without feedback would better support the stronger claim.
A2: Thank you for your suggestion, which helped us present the effects of feedback on intent prediction and behavioral compensation more clearly in the paper.
① In our architecture, VLM hallucinations primarily manifest as incorrect cognitive intents.
To assess whether behavioral feedback improves intent prediction itself, we compare DualWM with a variant that removes bottom-up behavioral-state feedback while retaining top-down cognitive conditioning. As shown in Table R2, adding feedback improves intent accuracy from 74.80% to 83.98% when both variants are evaluated on the same validation segments. This indicates that our lower-level behavioral dynamics representation provides useful feedback to the higher-level cognitive intent representation, helping reduce intent prediction errors.
② We also evaluate the behavioral layer's ability to compensate for incorrect cognitive guidance. Compensation denotes the proportion of cases in which the behavioral intent derived from the predicted trajectory is correct, given that the cognitive intent is incorrect. Adding feedback increases this proportion from 69.23% to 73.08%, indicating that DualWM has a stronger ability to compensate for incorrect cognitive guidance at the behavioral level.
③ Finally, to further assess the effect of behavioral feedback on closed-loop driving safety, we measure the proportions of routes with collisions and traffic violations. Adding feedback reduces the proportion of routes with collisions from 35.00% to 30.15% and the proportion with traffic violations from 20.00% to 14.85%, further supporting the contribution of behavioral feedback to safer driving. The closed-loop safety analysis is provided in Appendix D.3 (lines 1483–1487). The results are shown below.
Table R2. Effect of bottom-up feedback on intent prediction and driving performance.
| Variant | Intent accuracy ↑ | Compensation ↑ | Collision ↓ | Violation ↓ |
|---|---|---|---|---|
| w/o feedback | 74.80 | 69.23 | 35.00 | 20.00 |
| DualWM | 83.98 | 73.08 | 30.15 | 14.85 |
We have revised the manuscript and added the feedback experiment results to Table 13 in Appendix D.3 (lines 1513–1521).
W3: The reported average latency does not fully characterize the system’s responsiveness. Although Appendix C.4 explains how inference cost is measured, it remains unclear whether behavioral control continues during VLM inference and how much latency increases on query steps. Clarifying the execution schedule and reporting query-step versus nonquery-step latency would help assess the practical benefit of the asynchronous design.
A3: Thank you for the opportunity to clarify this point, which also helps us explain more clearly how we evaluate the practical benefits of our asynchronous design.
To clarify how behavioral control operates during VLM inference, we supplemented the standard synchronous evaluation with a latency-injection experiment. Standard synchronous evaluation assesses the method's performance under idealized timing conditions without inference-delay effects, whereas latency injection incorporates the measured inference time into simulation time, more closely reflecting real-world driving conditions. We also introduce an asynchronous execution mechanism to mitigate the effects of long inference delays during policy deployment. Below, we first describe the execution schedule in the standard synchronous evaluation, followed by DualWM's asynchronous execution mechanism and its latency-injection evaluation results.
A3 (continued):
① In the standard synchronous Bench2Drive/CARLA evaluation used for DualWM and the baselines, behavioral control does not continue while the VLM is running. Instead, the next control command is generated only after VLM inference finishes. CARLA also waits for the model output before advancing the simulation. Thus, the simulated environment does not advance during this wait, and the wall-clock inference time does not affect driving performance in this setting.
On the A800, query steps have a mean latency of 1401.16 ms and a p95 of 1477.97 ms, while non-query steps have a mean latency of 167.23 ms and a p95 of 176.06 ms. The latency distributions on both GPUs are shown below.
Table R3. Latency statistics of DualWM on the A800 and H20 at K=10 (ms).
| GPU | Scope | Mean | Median | P95 | Max |
|---|---|---|---|---|---|
| A800 | Query step | 1401.16 | 1389.58 | 1477.97 | 2230.83 |
| A800 | Non-query step | 167.23 | 167.28 | 176.06 | 232.99 |
| A800 | All steps | 291.66 | 167.99 | 1390.28 | 2230.83 |
| H20 | Query step | 1077.28 | 1053.61 | 1128.02 | 1767.83 |
| H20 | Non-query step | 164.26 | 164.17 | 167.43 | 182.83 |
| H20 | All steps | 257.07 | 164.63 | 1053.83 | 1767.83 |
We have revised the manuscript and added these latency statistics to Table 9 in Appendix D.1 (lines 1363–1380).
These results show that query-step latency is substantially higher than non-query-step latency. In real-world driving, however, the environment continues to evolve during inference, and long tail latencies can block control updates, limiting real-time deployment.
② To reduce the impact of long tail latencies on real-time deployment of closed-loop driving policies, our method uses DualWM's asynchronous dual-layer architecture during inference, allowing behavioral control to continue while the VLM is running. Until the VLM returns the intent features for the current query, the previous intent features remain in use, while the behavioral layer can continue adjusting the driving plan as it awaits the cognitive update. Because the intent features provide high-level guidance that remains relatively stable over a longer time window, they can still effectively support current action generation.
Under this asynchronous execution scheme, VLM queries run in parallel with the behavioral layer, and their cognitive outputs can be reused across multiple control steps, allowing behavioral control to continue while a cognitive update is pending. In the latency-injection evaluation, control-generation times remain broadly comparable across steps, averaging 182.41 ms per step, close to the 167.23 ms measured for non-query steps in the synchronous evaluation. This reduces the blocking effects of query-step tail latency on control updates. The latency distribution is shown in Table R4 below.
Table R4. Control-generation latency in the latency-injection evaluation (ms).
| GPU | Scope | Mean | Median | P95 | Max |
|---|---|---|---|---|---|
| A800 | All steps | 182.41 | 174.50 | 232.70 | 349.09 |
We have revised the manuscript and added the control-generation latency results and analysis to Appendix D.2, with the results reported in Table 10 (lines 1420–1424).
③ To further evaluate driving performance under inference delays with this asynchronous execution mechanism, we added a latency-injection experiment using measured A800 delays. In this setting, the simulated world continues advancing while outputs are pending, and the vehicle holds its previous control command until a new one becomes available, to simulate real-world driving conditions.
Under the same latency-injection setting, DualWM achieves 71.06 DS, compared with 35.36 for ORION, 35.90 for MindDrive, and 18.40 for DriveMoE. Relative to their respective scores without latency injection, ORION, MindDrive, and DriveMoE show DS reductions of 54.52%, 54.00%, and 75.21%, respectively. DualWM retains 87.25% of its original DS, a reduction of 12.75%. These results show that DualWM maintains better closed-loop driving performance than the other tested methods when inference delays are taken into account. The results are shown below.
Table R5. Closed-loop results under measured inference latency injection on Bench2Drive.
| Method | DS ↑ | SR (%) ↑ | RC (%) ↑ | Mean abilities (%) ↑ |
|---|---|---|---|---|
| DriveMoE | 18.40 | 0.45 | 58.61 | 3.41 |
| ORION | 35.36 | 9.09 | 72.55 | 18.05 |
| MindDrive | 35.90 | 8.64 | 70.46 | 16.74 |
| DualWM | 71.06 | 42.27 | 93.43 | 47.42 |
We have revised the manuscript and added these results and analyses to Table 4 in the main text (lines 399–414) and Appendix D.2 (lines 1383–1395).
We sincerely thank you for your careful, technically detailed review and constructive suggestions. We also appreciate your recognition of the motivation for separating persistent maneuver intent from rapidly evolving behavioral dynamics, our probabilistic formulation, and our closed-loop results. Over the approximately six weeks since submission, we have continued to improve our work and completed several additional experiments. We were particularly encouraged to find that the improvements we had been pursuing closely aligned with several of your suggestions. Your suggestions also helped us examine questions we had not previously investigated systematically and further strengthen the revised manuscript. Below, we address your comments point by point, supported by the relevant experimental results. Tables in this response are numbered independently as R1–R6. All manuscript section, table, and line references refer to the revised manuscript.
Critical Weaknesses
A1: Thank you for your detailed analysis and this targeted suggestion for a controlled comparison, which helps us examine the contribution of cognitive dynamics modeling more directly. The original cognitive-layer ablation also removed intent gating, while the prediction ablation disabled predictive modeling in both layers. These comparisons therefore assessed broader component contributions.
To examine the role of cognitive dynamics more directly, we refined the w/o cognitive dynamics control by replacing learned cognitive dynamics with cached VLM features while retaining the behavioral model, intent gating, and MoE planner. This control achieves 78.57 DS and 56.82% SR, compared with 81.44 DS and 60.45% SR for the full DualWM model. With the behavioral model and planner retained, the full model improves DS by 2.87 points and SR by 3.63 percentage points. These results support the additional benefit of learned cognitive dynamics beyond reusing cached VLM features. The results are shown below.
Table R1. Ablation on Bench2Drive. ΔDS is the DS change from the full model.
We have revised the manuscript and added these results and the control settings to Table 5 in the main text (lines 433–445) and Appendix D.3 (lines 1428–1440).