acceptodds
Under review as a conference paper at ICLR 2027

WAM-Diff2: Hierarchical AR-to-Diffusion Distillation for Highly Efficient Autonomous Driving VLA

Abstract

Vision-Language-Action (VLA) models have emerged as a prominent paradigm for end-to-end autonomous driving; however, their efficient deployment is severely constrained by high computational latency and exposure bias arising from sequential autoregressive decoding. Conversely, while specialized diffusion policies enable low-latency, parallel execution, training them from scratch typically yields narrow, single-task architectures that lack holistic visual-linguistic reasoning. Successfully transforming pre-trained autoregressive generalists into parallel diffusion models could combine multi-task cognitive intelligence with execution efficiency, yet this transition presents a formidable architectural challenge due to mismatched attention patterns (causal versus bidirectional) and divergent optimization objectives. To bridge this divide, we introduce WAM-Diff2, a multi-task discrete diffusion VLA framework powered by a three-stage hierarchical distillation strategy. By structuring the architectural shift through progressive block-wise adaptation, block-wise distillation, and model-wise cross-scale distillation, WAM-Diff2 preserves the underlying semantic foundations of the base model while accelerating inference. Extensive evaluations across driving understanding, perception, and planning benchmarks demonstrate that WAM-Diff2 effectively mitigates exposure bias and achieves performance parity with autoregressive baselines. Crucially, the autoregressive-to-diffusion transition yields a decoding speedup, which scales to an ultimate acceleration when combined with system-level optimizations including FlashInfer and CUDA Graphs.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-Language-Action Models, Block Diffusion Model, Knowledge Distillation, Autonomous Driving, Efficient Inference

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_5xz1OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper proposes a three-stage hierarchical distillation strategy: progressive block-wise adaptation, block-wise distillation, and model-wise cross-scale distillation (from an 8B diffusion teacher to a 2B student). For driving understanding, visual perception, and motion planning tasks, the authors first fine-tune Qwen3-VL-2B on multi-task driving data to obtain an AR driving VLA. They then convert this model into a discrete diffusion VLA with block-causal attention. This approach achieves an approximately 2.8× decoding speedup with only a small loss in performance; when combined with system-level optimizations (FlashInfer and CUDA Graphs), the cumulative acceleration reaches 15.1×.

Critical strengths
  1. The method is simple. It does not require any task-specific projection heads or additional models, and the decoding block size can be adjusted at runtime, enabling a flexible trade-off between latency and accuracy.
  2. The evaluation is comprehensive. A single checkpoint is tested across three types of tasks under the unified multi-task protocol, and the planning component is further evaluated in both open-loop (NAVSIM v1/v2) and closed-loop (Bench2Drive) settings under the task-specific protocol.
  3. The ablation study is informative, covering each distillation stage, the divergence objective (forward KL, reverse KL, JSD), and the teacher paradigm (AR teacher vs. diffusion teacher).
Critical weaknesses
  1. FlashInfer and CUDA Graphs are applied only to the diffusion model, but these optimizations could also be used for the AR baseline (both are commonly supported in vLLM and SGLang). A fairer comparison would be between an “optimized AR” model and an “optimized diffusion” model.
  2. Accuracy results are evaluated on the Ascend 910C NPU, whereas the hardware-specific inference optimizations were benchmarked on a CUDA-compatible platform, so the latency-accuracy trade-off is not verified on a single platform.
  3. Speculative decoding with parallel block drafters (e.g., DFlash) and its application to driving VLAs (e.g., FlashDrive) are cited in the related work, but no comparison is provided for any of the tasks.
  4. For motion planning, a commonly used alternative is a continuous action head (such as the flow-matching trajectory decoder used in Alpamayo-R1), which could be much faster: in the Alpamayo-R1 paper, flow-matching trajectory decoding takes 8.75 ms (5 steps) versus 222 ms for AR trajectory decoding (127 tokens). A discussion of this design is missing.
  5. There is no end-to-end profiling or comparison. The paper reports only per-token decoding latency without specifying the number of tokens per trajectory, the prefill and visual encoding latency, or the proportion of end-to-end latency attributed to decoding; consequently, the actual benefits for real-world deployment cannot be assessed.
  6. To my knowledge, prior work already exists on converting AR models into block diffusion models (e.g., Fast-dLLM v2, SDAR). The paper does not discuss how its strategy differs from these existing approaches.
  7. Table 4 shows that direct AR-to-diffusion adaptation followed by block-wise distillation yields 87.7 PDMS, lower than the 88.1 achieved by the AR baseline, and the score only reaches 88.3 when model-wise cross-scale distillation from the 8B diffusion teacher is added. A fair comparison would be to distill the 2B AR baseline from an 8B AR teacher under an on-policy setting. Similarly, the analysis of exposure bias (which shows a 5.8% reduction in average per-waypoint L2 error) appears to compare a distilled diffusion model against an AR model before distillation; therefore, it is difficult to determine whether the gains come from the paradigm transition or from the 8B teacher.
Questions for authors
  1. What is the decoding latency of the AR baseline when FlashInfer and CUDA Graphs are also applied, measured on the same platform as the diffusion model?
  2. How many tokens does a trajectory contain for the main results? Could the authors provide an end-to-end latency breakdown (visual encoding, prefill, and decoding) for both the AR baseline and the diffusion model?
  3. How does the proposed approach compare with applying speculative decoding (e.g., DFlash and FlashDrive) to the AR baseline, in terms of both latency and accuracy?
  4. How does the three-stage hierarchical distillation strategy differ from existing AR-to-block-diffusion conversion methods such as Fast-dLLM v2 and SDAR?
  5. How does the 2B AR baseline perform when distilled from an 8B AR teacher under an on-policy setting?
10 comments
AuthorsOpenReview ↗

Response to Reviewer 5xz1 1/5

We sincerely thank the reviewer for the detailed feedback. The suggested comparisons have helped us better substantiate our results and clarify our contribution. We respond to each point below.

W1&Q1:Comparison between optimized AR model and optimized Diffusion model

W1. FlashInfer and CUDA Graphs are applied only to the diffusion model, but these optimizations could also be used for the AR baseline (both are commonly supported in vLLM and SGLang). A fairer comparison would be between an “optimized AR” model and an “optimized diffusion” model.

Q1. What is the decoding latency of the AR baseline when FlashInfer and CUDA Graphs are also applied, measured on the same platform as the diffusion model?

1.What is the decoding latency of the AR baseline when FlashInfer and CUDA Graphs are also applied, measured on the same platform as the diffusion model?(Q1/W1)

We thank the reviewer for pointing out the need for a fair comparison. We applied the same system optimizations (FlashInfer and CUDA Graphs) to the AR baseline and evaluated both models on the same hardware platform.

ModelVision Encode (ms) ↓LLM Prefill (ms) ↓Decode (ms) ↓E2E (ms) ↓E2E P99 (ms) ↓
AR50.2380.381817.791948.982254.88
Diffusion49.0282.97670.11802.10892.14
AR + System Opt.31.7320.85474.82527.40579.16
Diffusion + System Opt.31.5121.28121.14173.83193.36

The results show that WAM-Diff2 achieves a 3.92× decoding speedup over the system-optimized AR baseline (121.14 ms vs. 474.82 ms).

AR and block-diffusion models benefit differently from system optimizations because of their distinct attention execution pattern. PyTorch's native SDPA already provides efficient support for standard causal attention in AR models, whereas custom block-causal attention masks in diffusion models can introduce additional Python/CUDA dispatch overhead. FlashInfer and CUDA Graphs help eliminate these execution bottlenecks, allowing WAM-Diff2 to better realize the computational benefits of parallel decoding. We will update these result to the revise manuscript.

W2:Latency-accuracy trade-off on single platform

W2. Accuracy results are evaluated on the Ascend 910C NPU, whereas the hardware-specific inference optimizations were benchmarked on a CUDA-compatible platform, so the latency-accuracy trade-off is not verified on a single platform.

Thank you for highlighting the importance of evaluating latency and accuracy on the same hardware platform.We evaluated the AR and WAM-Diff2 checkpoints reported in Table 2 of the main paper on a single Ascend 910C NPU, measuring decoding throughput, latency, and NAVSIM PDMS. Both models use the original PyTorch implementation with BF16 precision and batch size 1, without additional system optimizations.

ModelDecode Throughput (tokens/s) ↑Decode Latency (ms) ↓PDMS ↑
AR Baseline30.03279088.1
WAM-Diff2 (Ours)83.30100688.3

These same-platform results show a 2.77× decoding speedup while maintaining comparable planning performance (88.3 vs. 88.1 PDMS). The table summarizes representative operating points. Given the figure-upload limitations of this rebuttal response, we will included the full latency–accuracy trade-off curve in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer 5xz1 2/5

W5&Q2: End-to-end profiling&specifying the number of tokens per trajectory

W5. There is no end-to-end profiling or comparison. The paper reports only per-token decoding latency without specifying the number of tokens per trajectory, the prefill and visual encoding latency, or the proportion of end-to-end latency attributed to decoding; consequently, the actual benefits for real-world deployment cannot be assessed.

Q2. How many tokens does a trajectory contain for the main results? Could the authors provide an end-to-end latency breakdown (visual encoding, prefill, and decoding) for both the AR baseline and the diffusion model?

1.How many tokens does a trajectory contain for the main results?(W5/Q2)

Each trajectory consists of 8 future waypoints, serialized into 83.6 output tokens(mean count in Navtest).

2.Could the authors provide an end-to-end latency breakdown (visual encoding, prefill, and decoding) for both the AR baseline and the diffusion model?(W5/Q2)

Yes. We thank the reviewer for highlighting the importance of end-to-end latency evaluation. To provide a comprehensive latency analysis, we additionally profile visual encoding, LLM prefill, trajectory decoding, and total end-to-end (E2E) latency for both AR and WAM-Diff2 on a single NVIDIA H200 GPU with BF16 precision and batch size 1.

ModelVision Encode (ms) ↓LLM Prefill (ms) ↓Decode (ms) ↓E2E (ms) ↓E2E P99 (ms) ↓
AR50.2380.381817.791948.982254.88
Diffusion49.0282.97670.11802.10892.14
AR + System Opt.31.7320.85474.82527.40579.16
Diffusion + System Opt.31.5121.28121.14173.83193.36

The results show that decoding constitutes the primary latency bottleneck, accounting for 93.3% of the original AR model's E2E latency. Under comparable system optimizations (FlashInfer and CUDA Graphs), WAM-Diff2 reduces decoding latency from 474.82 ms to 121.14 ms, achieving a 3.92× decoding speedup over the optimized AR baseline. More importantly, this translates into a 3.03× E2E speedup (527.40 → 173.83 ms), with a P99 latency of 193.36 ms, meeting the 200-ms latency budget corresponding to 5 Hz under our benchmark setting.

AuthorsOpenReview ↗

Response to Reviewer 5xz1 3/5

W3&Q3:Comparison with speculative decoding method on both latency and accuracy

W3. Speculative decoding with parallel block drafters (e.g., DFlash) and its application to driving VLAs (e.g., FlashDrive) are cited in the related work, but no comparison is provided for any of the tasks.

Q3. How does the proposed approach compare with applying speculative decoding (e.g., DFlash and FlashDrive) to the AR baseline, in terms of both latency and accuracy?

1.How does the proposed approach compare with applying speculative decoding (e.g., DFlash and FlashDrive) to the AR baseline, in terms of both latency and accuracy?(Q3/W3)

We thank the reviewer for suggesting this comparison. We applied DFlash to our 2B AR baseline, using the same hardware platform and system optimizations (FlashInfer and CUDA Graphs), and evaluated both latency and planning accuracy. FlashDrive also uses DFlash for speculative reasoning; our experiment evaluates this component within our VLA setting.

ModelVision Encode (ms) ↓LLM Prefill (ms) ↓Decode (ms) ↓E2E (ms) ↓E2E P99 (ms) ↓PDMS
AR50.2380.381817.791948.982254.8888.14
AR + System Opt.31.7320.85474.82527.40579.1687.99
Diffusion + System Opt.31.5121.28121.14173.83193.3687.28
AR + System Opt. + Dflash31.7320.85134.96187.64279.5387.99

Numerical differences under system optimizations. FlashInfer employs optimized attention kernels with different floating-point reduction orders, which may introduce minor numerical variations. These differences can propagate through autoregressive or confidence-based diffusion decoding, potentially leading to variations in the generated outputs and evaluation metrics.

Under these matched conditions, WAM-Diff2 achieves lower latency across all three measures: decoding latency decreases from 134.96 to 121.14 ms, mean E2E latency from 187.64 to 173.83 ms, and E2E P99 from 279.53 to 193.36 ms. WAM-Diff2 achieves 87.28 PDMS, compared with 87.99 for DFlash.

The results show that speculative decoding substantially accelerates the AR baseline, reducing E2E latency from 527.40 ms to 187.64 ms. However, WAM-Diff2 achieves both lower decoding latency (121.14 vs. 134.96 ms) and lower E2E latency (173.83 vs. 187.64 ms).

More importantly, WAM-Diff2 achieves 30.8% lower P99 latency (193.36 vs. 279.53 ms), demonstrating more predictable inference latency. In contrast, DFlash can suffer from repeated speculation rejections and low acceptance rates in challenging scenarios, leading to substantial tail latency. Notably, WAM-Diff2 maintains its P99 below the 200-ms budget of a 5 Hz driving cycle, highlighting its advantage for latency-sensitive autonomous driving.

Furthermore, WAM-Diff2 performs inference with a single 2B model, without an auxiliary drafter or a separate autoregressive verification step(DFlash introduces an additional 283M parameters approximately 14% additional parameters relative to the 2B baseline, making on-vehicle deployment more demanding.). These results highlight WAM-Diff2’s advantages in tail latency and inference architecture, which is particularly valuable for latency-sensitive autonomous driving applications.

W4:Discussion about continuous action head

W4. For motion planning, a commonly used alternative is a continuous action head (such as the flow-matching trajectory decoder used in Alpamayo-R1), which could be much faster: in the Alpamayo-R1 paper, flow-matching trajectory decoding takes 8.75 ms (5 steps) versus 222 ms for AR trajectory decoding (127 tokens). A discussion of this design is missing.

Thank you for highlighting continuous action heads. Methods such as Alpamayo-R1 and MeanFuser achieve efficient trajectory generation through specialized decoders. However, separating action generation from the VLM introduces distinct optimization objectives and additional challenges in reasoning–action alignment. Recent VLAs, such as AutoVLA and LinkVLA, instead explore unified token-based generation to more tightly couple reasoning and actions.

WAM-Diff2 follows this direction, focusing on accelerating the shared token decoder rather than introducing a separate action head. Our hierarchical AR-to-diffusion distillation enables efficient parallel generation of trajectories, driving VQA, and structured perception outputs while preserving the pretrained VLA's multi-task capabilities. We will clarify this distinction in the revision.

Reviewer_SSEDOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces WAM-Diff2, a framework for converting autoregressive vision-language-action models into block-diffusion models for autonomous driving. It combines progressive block-wise adaptation, on-policy block-wise distillation, and optional cross-scale distillation to enable parallel decoding while retaining understanding, perception, and planning capabilities. Experiments across multiple benchmarks, including closed-loop driving, demonstrate approximately 2.8× decoding speedup without system-level optimizations, alongside task-dependent performance trade-offs.

Critical strengths
  1. Practical relevance. The paper converts autoregressive VLAs into block-diffusion models for unified understanding, perception, and planning, achieving approximately 2.8× decoding speedup without additional system-level optimizations.
  2. Broad evaluation. Experiments cover both unified multi-task models and task-specific models across multiple benchmarks, including closed-loop driving.
  3. Encouraging robust evidence. Prefix-perturbation experiments show lower sensitivity to erroneous trajectory prefixes than the AR baseline, suggesting potential for reducing error propagation.
Critical weaknesses
  1. The independent contribution of Stage I is unclear. Table 4 does not clarify whether “Direct AR-to-Diffusion” includes the complete progressive Stage I. It is therefore difficult to assess the necessity of progressive adaptation and distinguish its contribution from subsequent distillation.
  2. Candidate trajectory selection is insufficiently specified. The paper does not adequately describe the number of candidates, the scoring mechanism, or its required inputs, limiting reproducibility and assessment of the gains relative to the additional computational cost.
  3. The benefits of the additional training complexity are not fully justified. Despite the approximately 2.8× decoding speedup, the unified model declines on all five task metrics, and the closed-loop success rate and driving score also decrease. In Table 2, the task-specific diffusion planner shows only modest gains over its AR counterpart without candidate trajectory selection. Given the computational cost of progressive adaptation and multi-stage distillation, the incremental value over simpler conversion pipelines requires stronger justification.
  4. The on-policy design lacks an isolated ablation. Existing ablations compare training stages, losses, and teacher types, but not on-policy versus off-policy distillation under otherwise identical conditions. The prefix-perturbation results support the robustness of the overall model but do not isolate the contribution of on-policy training.
Questions for authors

See Weaknesses for details.

5 comments
AuthorsOpenReview ↗

Response to Reviewer SSED 1/3

We sincerely thank the reviewer for carefully examining our training framework and highlighting the need to better justify its additional complexity and clarify the contributions of individual stages. We have added targeted ablations and implementation details in response, as presented below.

W1: Isolate the contribution of Stage I

W1. The independent contribution of Stage I is unclear. Table 4 does not clarify whether “Direct AR-to-Diffusion” includes the complete progressive Stage I. It is therefore difficult to assess the necessity of progressive adaptation and distinguish its contribution from subsequent distillation.

Thank you for pointing out this ambiguity. “Direct AR-to-Diffusion” in Table 4 does not include progressive block-size adaptation. It directly increases the block size from 1 to 32 and trains at block size 32, using the same training-step budget as Stage I.

We have added a Progressive Block-wise Adaptation (Stage I) row to Table 4 to show the effect of progressive adaptation. On NAVSIM, Stage I achieves 87.2 PDMS, compared with 84.1 for Direct AR-to-Diffusion at a fixed block size. PDMS then increases to 87.7 after block-wise distillation and 88.3 after model-wise distillation. The expanded table shows the gains at each stage.

MethodNAVSIM PDMS ↑
AR Baseline88.1
Direct Conversion84.1
Progressive Block-wise Adaptation87.2
+ Block-wise Distillation87.7
+ Model-wise Distillation88.3

W2: Candidate trajectory selection is insufficiently specified

W2. Candidate trajectory selection is insufficiently specified. The paper does not adequately describe the number of candidates, the scoring mechanism, or its required inputs, limiting reproducibility and assessment of the gains relative to the additional computational cost.

Thank you for highlighting the need to clarify the candidate selection procedure. For the starred results in Table 2, we incorporate the scoring mechanism from DriveSuprim to enable comparison with scorer-based planning methods. Specifically, we generate 64 candidate trajectories per scene through parallel stochastic sampling with temperature scaling. Vision encoding and LLM prefill are computed once per frame, and the resulting prefix KV cache is shared across all 64 candidates.The scorer extracts scene features from the front, front-left, and front-right camera views using ResNet-34, while a two-layer MLP encodes the candidate trajectories into trajectory features. These features are fused through cross-attention to produce a score for each candidate. We select the highest-scoring trajectory as the final prediction.

We acknowledge that generate 64 candidate trajectories takes additional computational cost. Under system-optimized inference, generating 64 candidates takes 1,432.28 ms. Scoring requires an additional 29.4 ms, bringing the combined generation and scoring cost to 1,461.68 ms per scene. Our default configuration generates a single trajectory without a scorer, requiring only 173.83 ms E2E latency. We will explicitly report this additional cost alongside the starred results.

The scorer-assisted setting is included for comparison with methods such as DriveSuprim, rather than to position WAM-Diff2 as a specialized state-of-the-art motion planner. Our primary contribution is a hierarchical AR-to-diffusion conversion framework for multi-task VLAs.

AuthorsOpenReview ↗

Response to Reviewer SSED 2/3

W3: Clearify the benefits of the additional training complexity

W3.The benefits of the additional training complexity are not fully justified. Despite the approximately 2.8× decoding speedup, the unified model declines on all five task metrics, and the closed-loop success rate and driving score also decrease. In Table 2, the task-specific diffusion planner shows only modest gains over its AR counterpart without candidate trajectory selection. Given the computational cost of progressive adaptation and multi-stage distillation, the incremental value over simpler conversion pipelines requires stronger justification.

Thank you for raising this concern. To assess the value of the additional training stages, we expanded Table 4 with the standalone progressive adaptation result and two matched-step continued-SFT baselines. We compare direct conversion and progressive adaptation using the same S1 training steps. Starting from the Stage I checkpoint, we then continue SFT for S2 steps and a further S3 steps, matching the cumulative training steps of Stage II and the full pipeline, respectively.

MethodStage IIStage IIITraining StepsPDMS ↑
AR Baseline------88.1
Direct Conversion----S184.1
Progressive Only (Stage I)----S187.2
Progressive + Continued SFTSFT--S1+S287.3
Progressive + BWDBWD--S1+S287.7
Progressive + Continued SFTSFTSFTS1+S2+S387.3
Progressive + BWD + MWD (Ours)BWDMWDS1+S2+S388.3
Here, S1, S2, and S3 denote the numbers of student training updates in Stages I, II, and III, respectively.

Under the same Stage I budget, progressive adaptation improves PDMS from 84.1 to 87.2 (+3.1) over direct conversion. Continued SFT reaches 87.3 PDMS at both subsequent budgets. At the corresponding step counts, BWD achieves 87.7 (+0.4 over continued SFT), while the full BWD + MWD pipeline achieves 88.3 (+1.0 over continued SFT), recovering the task-specific AR baseline’s planning performance (88.1 PDMS).

We also extended the stage-wise ablation to the unified multi-task model. Direct conversion reduces DriveBench from 51.2 to 44.1, COCO mAP from 39.2 to 34.7, and NAVSIM PDMS from 88.14 to 84.04. Performance improves at every subsequent stage. Relative to direct conversion, the full hierarchy recovers 1.8 points on LingoQA, 4.7 points on DriveBench, 1.6 points on COCO mAP, and 3.40 points on NAVSIM PDMS.

MethodLingoQA ↑DriveBench ↑COCO mAP ↑NAVSIM PDMS ↑
AR Baseline68.451.239.288.14
Direct Conversion64.044.134.784.04
Progressive Only65.248.235.886.51
+ Block-wise Distillation65.448.636.187.09
+ Model-wise Distillation65.848.836.387.44

Together, these comparisons demonstrate the value of the additional training stages. The matched-step planning experiments show gains beyond simply extending SFT, while the multi-task ablation shows progressive recovery across language understanding, perception, and planning. The value of the hierarchy therefore extends beyond the task-specific PDMS gain: it substantially reduces the capability loss introduced by direct conversion in a unified VLA.

AuthorsOpenReview ↗

Response to Reviewer SSED 3/3

W4. isolate the contribution of on-policy training.

W4.The on-policy design lacks an isolated ablation. Existing ablations compare training stages, losses, and teacher types, but not on-policy versus off-policy distillation under otherwise identical conditions. The prefix-perturbation results support the robustness of the overall model but do not isolate the contribution of on-policy training.

Thank you for highlighting the need for isolated ablation. We conducted a controlled comparison between on-policy and off-policy distillation, keeping the teacher and student models, training configurations, and distillation objectives fixed. We evaluated both variants on NAVSIM planning performance and prefix-perturbation robustness.

On-policy distillation improves PDMS from 88.1 to 88.3, reduces ΔFDE from 0.467 to 0.458, and lowers P(Shift > 1.5 m) from 20.7% to 19.2%, a reduction of 1.5 percentage points. Clean FDE, final shift, and P90 shift also improve, as shown in the table.

Distillation StrategyPDMS ↑Clean FDE ↓Perturbed FDE ↓ΔFDE ↓Final Shift ↓P90 Shift ↓P(Shift > 1.5 m) ↓
Off-policy Distillation88.11.0141.4810.4671.181.9120.7%
On-policy Distillation (Ours)88.30.9981.4560.4581.161.8519.2%

These results demonstrate an incremental contribution from on-policy distillation to both planning performance and perturbation robustness under the controlled comparison.

Reviewer_6FNKOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents WAM-Diff2, which converts a Qwen3-VL-based autoregressive driving model into a block-wise discrete diffusion model for faster generation. Training gradually increases the decoding block size. Smaller-block diffusion teachers guide larger-block students on states generated by the students. The method can also distill an 8B diffusion model into a 2B model. Experiments cover driving question answering, COCO grounding, NAVSIM planning, and Bench2Drive closed-loop driving. Generation becomes faster, while performance on most tasks stays close to the original model. Errors also grow less when the trajectory prefix is perturbed.

Critical strengths

The paper addresses a practical deployment problem and checks planning, perception, and language performance. Evaluating several tasks with the same checkpoint is useful. The paper also clearly distinguishes this evaluation from evaluation tailored to a single planning task. Without system optimizations, block-32 decoding raises throughput from 44.5 to 124.8 tokens/s, while NAVSIM v1 PDMS drops from 88.14 to 87.44. The controlled prefix-perturbation experiment also gives a direct way to examine how sensitive the model is to earlier trajectory errors.

Critical weaknesses
  1. In Table 1, the autoregressive model runs at 44.5 tokens/s. The diffusion model runs at 124.8 tokens/s without FlashInfer and CUDA Graphs, and 673.4 tokens/s with them. The paper clearly separates the roughly 2.8x gain from model conversion and the final 15.1x gain. However, it does not report an autoregressive baseline with comparable system optimizations. Appendix A.3 only describes the hardware as a CUDA-compatible platform. The accelerator, precision, inference batch size, and timing workload are missing. The paper also does not report image encoding, prompt prefill, complete trajectory-generation latency, or tail latency. Decoding throughput alone does not show how much faster the full system would be in use.
  2. In Table 4, PDMS rises from 84.1 after direct conversion to 87.7 with block-wise distillation and 88.3 with model-wise distillation. But the paper does not compare against gradually increasing the block size without distillation, or direct conversion with the same training budget. Each block size and distillation stage receives five epochs, so some of the gain may come from additional training. The paper discusses earlier block-size schedules and autoregressive-to-diffusion methods, but does not compare implementations under matched conditions. Reporting training costs is helpful, but a matched-budget comparison is still needed.
  3. In Table 8, adding a 1 m perturbation to the first two ground-truth waypoints produces final displacements of 1.61 m for the autoregressive model and 1.16 m for WAM-Diff2. Appendix A.5 also reports a 5.8% reduction in average waypoint error over 12,146 paired samples. These are positive results. However, the models differ in training procedure as well as attention structure, and teacher supervision may also differ. The experiment therefore does not show whether the improvement comes from bidirectional decoding, training on the model’s own generated states, or additional distillation. Tests with several perturbation sizes and errors from naturally generated prefixes would help.
  4. In the unified multi-task experiment, block-32 diffusion reduces DriveBench from 51.23 to 48.80, LingoQA from 68.40 to 65.80, and COCO mAP from 39.20 to 36.30. Bench2Drive success rate falls from 51.96% to 49.55%, and driving score falls from 80.51 to 78.93. These losses may be acceptable in exchange for faster inference, but the claim of no meaningful performance degradation needs a clear tolerance and uncertainty estimates. The training-stage ablations are also limited to NAVSIM, and conversion is tested only within the Qwen3-VL family. The conclusions should stay within the models and tasks that were tested.
Questions for authors

Q1. What hardware and settings were used for timing? How much faster is WAM-Diff2 than an autoregressive model with comparable system optimizations? Please report latency from image input to a complete trajectory, including its distribution. Q2. Can you compare progressive block-size increases, block-wise distillation, and model-wise distillation with matched training steps or compute? Please include a baseline that changes block size without the proposed distillation hierarchy. Q3. Does the perturbation advantage remain with matched teacher supervision and training budgets, including an autoregressive model trained on its own generated prefixes? Q4. Can you report uncertainty for the multi-task and Bench2Drive results, add multi-task ablations for each training stage, and state how much accuracy loss is considered acceptable for the speed gains?

6 comments
AuthorsOpenReview ↗

Response to Reviewer 6FNK 1/5

We sincerely thank the reviewer for the careful review and constructive suggestions on evaluation fairness, robustness, and the accuracy–efficiency trade-off. These comments have helped us strengthen the experimental evidence and clarify the scope of our conclusions. We address each concern below.

W1/Q1: Fairness of Speed Comparisons and Practical End-to-End Latency

  1. W1. In Table 1, the autoregressive model runs at 44.5 tokens/s. The diffusion model runs at 124.8 tokens/s without FlashInfer and CUDA Graphs, and 673.4 tokens/s with them. The paper clearly separates the roughly 2.8x gain from model conversion and the final 15.1x gain. However, it does not report an autoregressive baseline with comparable system optimizations. Appendix A.3 only describes the hardware as a CUDA-compatible platform. The accelerator, precision, inference batch size, and timing workload are missing. The paper also does not report image encoding, prompt prefill, complete trajectory-generation latency, or tail latency. Decoding throughput alone does not show how much faster the full system would be in use.

  2. Q1. What hardware and settings were used for timing? How much faster is WAM-Diff2 than an autoregressive model with comparable system optimizations? Please report latency from image input to a complete trajectory, including its distribution.

1. Hardware and settings used for timing?W1/Q1W1/Q1

Thank you for the question. WAM-Diff2 uses a single H200 GPU , BF16 precision, batch size 1 for timing experiments. We conduct timing experiments on the NAVSIM navtest split, with approximately 1,980 visual tokens and 200 text tokens per input, generating an average of 83.6 output tokens per sample.

2. Compare WAM-Diff2 with autoregressive baseline with comparable system optimizations?W1/Q1W1/Q1

Thank you for raising comparision , we apply comparable system optimizations to the AR baseline and report a detailed latency breakdown as shown in Table. Under comparable system optimizations, WAM-Diff2 achieves a 3.92× decoding speedup 4748˙2→1211˙4ms474\.82 → 121\.14 ms and a 3.03× E2E speedup 5274˙0→1738˙3ms527\.40 → 173\.83 ms over the optimized AR baseline. We will update these result in the revise manuscript.

ModelVision Encode msms ↓LLM Prefill msms ↓Decode msms ↓E2E msms ↓E2E P99 msms ↓
AR50.2380.381817.791948.982254.88
Diffusion49.0282.97670.11802.10892.14
AR + System Opt.31.7320.85474.82527.40579.16
Diffusion + System Opt.31.5121.28121.14173.83193.36

Impact of system optimizations. PyTorch's native SDPA efficiently supports standard causal attention in AR models, whereas block-causal attention introduces additional masking and execution overhead. FlashInfer and CUDA Graphs effectively reduce these overheads, allowing WAM-Diff2 to better exploit its inherent parallel decoding capability. Consequently, system optimizations provide greater benefits to diffusion decoding, increasing its speedup over AR from 2.71× to 3.92× and its end-to-end speedup from 2.43× to 3.03×.

3. Report latency from image input to a complete trajectory, including its distribution. (Q1)

As show in Table, decoding accounts for over 90% of the optimized AR baseline's E2E latency. WAM-Diff2 substantially reduces this bottleneck through parallel decoding. Confidence-based decoding τ=0.9\tau=0.9 with a maximum of 12 steps effectively bounds decoding iterations and controls tail latency. Consequently, WAM-Diff2 achieves a tightly concentrated latency distribution, with a mean of 173.83 ms and P99 of 193.36 ms, both within the 200-ms latency budget corresponding to 5 Hz, which is particularly important for timely decision-making and safety-critical autonomous driving applications.

AuthorsOpenReview ↗

Response to Reviewer 6FNK 2/5

W2/Q2: Fairness of Training Budget and Effectiveness of Hierarchical Distillation

W2. In Table 4, PDMS rises from 84.1 after direct conversion to 87.7 with block-wise distillation and 88.3 with model-wise distillation. But the paper does not compare against gradually increasing the block size without distillation, or direct conversion with the same training budget. Each block size and distillation stage receives five epochs, so some of the gain may come from additional training. The paper discusses earlier block-size schedules and autoregressive-to-diffusion methods, but does not compare implementations under matched conditions. Reporting training costs is helpful, but a matched-budget comparison is still needed.

Q2. Can you compare progressive block-size increases, block-wise distillation, and model-wise distillation with matched training steps or compute? Please include a baseline that changes block size without the proposed distillation hierarchy.

1.Compare each training stage with matched training stepsW2/Q2W2/Q2

We have expanded the table to distinguish these effects from the benefits of distillation.

Effectiveness of progressive adaptation under matched training steps. Direct conversion and progressive block-wise adaptation both use S1 training steps. Progressive adaptation achieves 87.2 PDMS, compared with 84.1 for direct conversion at a fixed target block size. This 3.1-point improvement demonstrates the benefit of the progressive block-size schedule under the same training-step budget.

Effectiveness of hierarchical distillation . Starting from the Stage I checkpoint, we continued SFT for S2 steps, then for a further S3 steps, matching the cumulative training steps of Stage II and the full pipeline, respectively. Both continued-SFT checkpoints achieve 87.3 PDMS, indicating that continued SFT has plateaued over the training budgets. At the corresponding step counts, block-wise distillation achieves 87.7 PDMS +04˙+0\.4, while block-wise followed by model-wise distillation achieves 88.3 PDMS +10˙+1\.0. These comparisons show that hierarchical distillation provides gains beyond those obtained by simply extending SFT.

Here, S1, S2, and S3 denote the numbers of student training updates in Stages I, II, and III, respectively.

MethodStage IIStage IIITraining StepsPDMS ↑
AR Baseline------88.1
Direct Conversion----S184.1
Progressive Only (Stage I)----S187.2
Progressive + Continued SFTSFT--S1+S287.3
Progressive + BWDBWD--S1+S287.7
Progressive + Continued SFTSFTSFTS1+S2+S387.3
Progressive + BWD + MWD (Ours)BWDMWDS1+S2+S388.3
AuthorsOpenReview ↗

Response to Reviewer 6FNK 3/5

W3/Q3: Fairness of Robustness Comparison and Sources of Improvement

W3. In Table 8, adding a 1 m perturbation to the first two ground-truth waypoints produces final displacements of 1.61 m for the autoregressive model and 1.16 m for WAM-Diff2. Appendix A.5 also reports a 5.8% reduction in average waypoint error over 12,146 paired samples. These are positive results. However, the models differ in training procedure as well as attention structure, and teacher supervision may also differ. The experiment therefore does not show whether the improvement comes from bidirectional decoding, training on the model’s own generated states, or additional distillation. Tests with several perturbation sizes and errors from naturally generated prefixes would help.

Q3. Does the perturbation advantage remain with matched teacher supervision and training budgets, including an autoregressive model trained on its own generated prefixes?

1.Does the perturbation advantage remain with matched teacher supervision and training budgets, including an autoregressive model trained on its own generated prefixes? Q3Q3

Yes. We added two AR baselines to account for additional training and teacher supervision: ii continued SFT with a matched training budget, and iiii on-policy distillation OPDOPD of the 2B AR baseline using an 8B AR teacher, with the student trained on its own generated prefixes.

ModelPDMS ↑ΔFDE ↓Final Shift ↓P90 Shift ↓PShift 15˙mShift \> 1\.5 m ↓
AR Baseline88.1+0.7591.613.3036.8%
AR + Continue SFT (Match Budget)87.8+0.7291.573.2334.6%
AR + OPD (8B AR Teacher)88.4+0.6031.392.4226.6%
WAM-Diff2 (Ours)88.3+0.4581.161.8519.2%

Matched-budget continued SFT modestly improves robustness, reducing ΔFDE from 0.759 to 0.729, while AR+OPD further reduces it to 0.603. Nevertheless, WAM-Diff2 achieves the lowest perturbation sensitivity across all four robustness metrics. Notably, although AR+OPD achieves a slightly higher PDMS **88\.4 vs\. 88\.3**, WAM-Diff2 reduces ΔFDE by 24.0% ∗∗06˙03→04˙58∗∗**0\.603 → 0\.458** and final shift by 16.5% ∗∗13˙9→11˙6∗∗**1\.39 → 1\.16** relative to this stronger AR baseline.

We also extended the average per-waypoint L2 error analysis in Appendix A.5 to AR+OPD. The original 5.8% reduction was measured against the undistilled AR baseline; WAM-Diff2 retains a 3.6% reduction against AR+OPD (0.5797 → 0.5589 m). This demonstrates that its waypoint-error advantage also persists after strengthening the AR baseline with 8B-teacher supervision and on-policy training.

These results demonstrate that WAM-Diff2’s robustness advantage persists against both matched-budget continued SFT and an on-policy-distilled AR model trained on its own generated prefixes.

2.Tests with several perturbation sizes and errors from naturally generated prefixes .(W3)

We also extended Table 8 to include multiple perturbation magnitudes and errors derived from naturally generated trajectory prefixes.

PerturbationAR ΔFDE ↓AR Shift ↓Diff ΔFDE ↓Diff Shift ↓
0.5 m0.3720.990.2200.63
1.0 m0.7591.610.4581.16
1.5 m1.4782.421.0761.83
Natural AR Prefix error0.3571.030.3440.874
Natural Diffusion Prefix error0.3310.9710.3080.809

Across all perturbation magnitudes, WAM-Diff2 consistently exhibits lower ΔFDE and final displacement. This advantage also persists when perturbations are derived from naturally generated AR and diffusion prefixes, further supporting its robustness under more realistic error patterns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.