acceptodds
Withdrawn from ICLR 2027

Scene-Adaptive Readout and Decoding in Diffusion Driving VLAs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Diffusion driving vision-language-action models repeatedly execute the backbone at every denoising stage, incurring substantial inference cost, while it remains unclear whether all scenes require the same decoding computation. To distinguish when a trajectory becomes readable from how much further refinement it requires, we analyze trajectory readout across network depth and decoding progress. We identify a depth-decoding dissociation in which late-layer representations already support useful trajectory readout after the first forward sweep, whereas early-layer representations benefit more from continued decoding. We further find that both the preferred readout layer and decoding depth vary across scenes. Based on these observations, we propose a scene-adaptive readout and decoding framework. A scene-conditioned readout MoE adaptively weights and combines trajectory predictions from layer-specific late-layer representations after a single backbone sweep, while an adaptive decoding policy applies additional native refinement only when required by the scene. We evaluate our framework on two diffusion driving VLAs across the Waymo E2E and nuScenes benchmarks. Our method improves validation RFS by up to 0.56 while also improving test performance, and achieves an end-to-end speedup of up to 6.0 on Waymo E2E, while consistently reducing trajectory error on nuScenes. These results demonstrate that scene-level readout and decoding allocation improves planning quality while avoiding unnecessary decoding.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Diffusion Language Models, Vision Language Action Models, Autonomous Driving, Representation Analysis, Adaptive Computation, Efficient Inference

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_2aLtOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a scene-adaptive inference framework to improve the efficiency and planning performance of diffusion-based driving vision-language-action (VLA) models. The authors analyze trajectory readout quality across network depth and decoding stages, observing that intermediate-to-late hidden representations can support accurate trajectory prediction after a single forward pass. Based on these observations, the proposed framework combines a scene-adaptive readout Mixture-of-Experts (MoE) with an adaptive decoding policy that selectively performs additional refinement. Experiments on two diffusion driving VLAs demonstrate substantial inference speedups while improving trajectory planning performance.

Critical strengths

S1. Significant inference efficiency improvements. The proposed method achieves substantial end-to-end speedups of up to 6.03×6.03\times while preserving or improving trajectory planning performance. These improvements are particularly meaningful given the high computational cost of iterative decoding in diffusion-based driving VLAs.

S2. Interesting analysis of trajectory representations. The analysis of trajectory readout across network depth and decoding stages is interesting. In particular, the observation that useful trajectory information can be extracted from intermediate-to-late representations after the first forward pass provides a compelling motivation for reducing unnecessary diffusion decoding.

S3. Evaluation across different diffusion VLA architectures. The improvements on two different diffusion driving VLAs, together with evaluations on Waymo E2E and nuScenes, support the practical applicability of the proposed approach.

Critical weaknesses

W1. Limited justification for the complexity of the proposed framework. The method combines multiple components, including layer-specific readout heads, latent editors, a scene-adaptive MoE, an auxiliary readout branch, and an adaptive decoding policy. While the overall framework achieves strong empirical results, the motivation and necessity of several individual components are not sufficiently clear. Although component-wise ablations are provided, a more principled explanation of the overall design would strengthen the methodological contribution.

W2. Insufficient explanation of trajectory readout behavior across decoding stages. One of the most interesting observations is that the first forward pass achieves the highest trajectory readout RFS for a substantial proportion of scenes, while additional decoding can sometimes reduce performance. However, the underlying reason for this behavior remains unclear. In particular, it is difficult to determine whether the degradation results from changes in trajectory representations during iterative decoding or from the use of separately trained, stage-specific supervised probes. Further analysis would help clarify whether the observed behavior reflects an intrinsic property of diffusion decoding or the readout evaluation procedure.

W3. Unclear necessity of scene-adaptive layer selection. The paper motivates scene-adaptive readout through variation in the oracle-optimal layer across scenes. However, Appendix Table A2 shows that L27 receives the largest mixture weight for the vast majority of scenes. This raises the question of whether scene-conditioned fusion offers a substantial advantage over a simpler fixed-layer readout. Comparisons against fixed-layer readout and simple non-adaptive fusion strategies would help establish the contribution of the proposed MoE.

Questions for authors

Q1. Role of the one-pass draft in early trajectory readout. Equation (5) uses both intermediate-layer hidden representations and the one-pass draft obtained from the final LM head. If intermediate representations already contain sufficient information for accurate trajectory prediction, what additional benefit does the one-pass draft provide? An ablation comparing hidden-state-only, draft-only, and combined readout would help clarify its role.

Q2. Rationale for the adaptive decoding stopping criterion. The decoding gate activates additional refinement when the initial readout is predicted to benefit from further decoding. However, the refinement process terminates when the intermediate prediction becomes sufficiently close to the initial readout. Why should agreement with the initial prediction indicate that further refinement is unnecessary, particularly when that prediction was considered insufficient?

Q3. Endpoint-based trajectory agreement. Equation (15) determines the stopping point using only the distance between the final predicted waypoints. Since two trajectories can have similar endpoints while following substantially different paths, what motivates using endpoint agreement rather than a full-trajectory discrepancy? An ablation or sensitivity analysis would help justify this design choice.

Reviewer_XyhROpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper tackles the problem of making decoding in diffusion-based driving VLA models adaptive rather than fixed in the number of forward-passes and choice of readout layers. The approach to adaptive decoding described in Section 4 is motivated by the analysis of the usefulness of early-forward-pass and varying-layer representations performed in Section 3. Section 5 presents experiments on WOD-E2E and nuScenes, focusing on the improvements of the adaptive approach over dVLM-AD and and Fast-dDrive (the diffusion-based methods in particular), while providing AR-based numbers for reference. The adaptive approach provides substantial speedups over diffusion VLA baselines, but at the cost of significant architectural complexity.

Critical strengths
  • The paper is right to start with an empirical analysis of existing methods to motivate its approach, although I believe that no generalizable insights have been extracted from their empirical analysis.

  • The speedups achieved by the final approach (but also the +Readout only ablation) over the baselines are substantial.

Critical weaknesses

At the high level, this paper suffers from many hard-coded architectural decisions which have not been ablated, and which are based on an empirical analysis of two prior works (dVLM-AD and and Fast-dDrive):

  • In Section 3 (Trajectory Analysis), the authors omit the fact that dVLM-AD and Fast-dDrive use different respective backbones (LLaDA-8B versus Fast-dVLM which have 32 versus 36 text layers, respectively). So the fact that the stable readout band performs best for L21-L29 for both these distinct architectures sounds accidental and is not clarified.

  • The “stable readout band” is motivated by an RFS improvement of ~0.1, considering that the y-axis ranges from 7.9 to 8.0 RFS. This is hardly a meaningful enough difference to motivate hardcoding for the entire method.

  • “Overall, L25 and the first forward sweep provide global defaults” (line 242) is a very ad-hoc design choice that has no explanatory adequacy for why this layer works best or whether it extrapolates to other backbones.

The architecture developed in Section 4 is simply proposed as optimal and is hardly ablated, despite introducing a huge number of moving parts simultaneously:

  • In Section 4.2, there is no good reason to introduce a scene-adaptive readout only for the readout band identified in Section 3. Extending the readout to every layer would provide a learnable way for the model to select whatever layer it deems most useful without hard-coding.

  • I don’t think the MoE described in Equation (9) is an MoE in the proper sense of the term. Using a separate FFN wℓw_\ell to process each individual layer’s representation, without any FFN-internal routing, does not make the described ensemble of FFNs an MoE.

  • The MoE described above introduces variance, and lines 333-335 and equation (11) introduce yet another hard-coded readout branch based on layer 27 to stabilize the variance of the MoE prediction (no ablations have been reported).

The latency gains in Table 1 are impressive, but the ablation of Table 3 shows that the +Readout alone is sufficient to achieve the entirety of the speedup — meanwhile, since the RFS and ADE measurements are provided without random seeds, the marginal gains on these metrics from the additional +Scene-Adaptive MoE and +Scene-Adaptive Decoding components cannot be viewed as statistically significant. This suggests that the Early Trajectory Readout alone is responsible for virtually all the meaningful latency gain.

Overall, while I agree with the necessity of an empirically-driven approach to adaptive VLA inference, this paper reads much more like a systems report than as a generalizable research result. I recommend that the authors establish what RFS gains and drops are statistically significant, and cut out the architectural components that do not decrease latency and do not improve quality by a statistically significant margin.

Questions for authors
  • Why does the analysis in Section 3 treat both dVLM-AD and Fast-dDrive as if their original implementations both had 32 layers? Are you swapping in the same LLaDA-8B backbone for both approaches including Fast-dDrive?
Reviewer_cpwzOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes an adaptive accleeration method for diffusion driving VLAs with trained readout from intermidiate hidden states. The authors observe that useful information can already be extracted from the first forward pass in diffusion process and train a scene-adaptive MoE to combine prediction from hidden features to determine whether further refinement is needed. The method achieves relatively strong inference speedup on Waymo E2E and nuScenes.

Critical strengths
  1. The paper shows that useful trajectory information can be extracted from intermediate hidden states within the first forward pass. The authors train a mixture of expert over different layers readout instead of leveraging the signal in a training-free way, which makes the method seems more promising. Though I think more analysis on this can be interesting.
  2. The method achieves substantial speedup while improving trajectory prediction quality. The practical benefits seems meaningful.
Critical weaknesses
  1. The evidence for the benefit of scene-adaptive readout seems limited. According to Table 3, a simple supervised readout already achieves most of the speedup and performance improvement, while the additional MoE brings relatively marginal gains.
  2. The understanding on how the MoE readout works is limited. Table A2 shows that the learned MoE assigns the largest weight to L27, even though the paper motivates its aprroach by showing that different scenes prefer different layers. A qualitative study on the specific scene that each layer contributes might be interesting and brings more insights on how to further improve the method.
Questions for authors

Same as weakness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.