acceptodds
Withdrawn from ICLR 2027

OneDrive: Unified Multi-Paradigm Driving with Vision-Language-Action Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-Language Models (VLMs) excel at autoregressive text generation, yet end-to-end autonomous driving requires multi-task learning with structured outputs and heterogeneous decoding behaviors, such as autoregressive language generation, parallel object detection and trajectory regression. To accommodate these differences, existing systems typically introduce separate or cascaded decoders, resulting in architectural fragmentation and limited backbone reuse. In this work, we present a unified autonomous driving framework built upon a pretrained VLM, where heterogeneous decoding behaviors are reconciled within a single transformer decoder. We demonstrate that pretrained VLM attention exhibits strong transferability beyond pure language modeling. By organizing visual and structured query tokens within a single causal decoder, structured queries can naturally condition on visual context through the original attention mechanism. Textual and structured outputs share a common attention backbone, enabling stable joint optimization across heterogeneous tasks. Trajectory planning is realized within the same causal LLM decoder by introducing structured trajectory queries. This unified formulation enables planning to share the pretrained attention backbone with images and perception tokens. Extensive experiments on end-to-end autonomous driving benchmarks demonstrate state-of-the-art performance, including 0.28 L2 and 0.18 collision rate on nuScenes open-loop evaluation and competitive results (87.3 PDMS) on NAVSIM closed-loop evaluation. The full model preserves multi-modal generation capability, while an efficient inference mode achieves approximately 40% lower latency.

Primary area:
applications to robotics, autonomy, planning
Keywords:
autonomous driving, vision language model, vision language action model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_GS2yOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes OneDrive, a unified VLM-based autonomous driving framework that integrates perception, planning, and text generation within a shared transformer decoder. It reuses pretrained causal attention and introduces task-specific FFNs and query self-attention for structured predictions. Experiments on nuScenes and NAVSIM demonstrate competitive planning performance and improved inference efficiency.

Critical strengths

The paper presents a unified framework for heterogeneous driving tasks within a single decoder. The integration is technically reasonable. The staged-training experiments show improvements within the proposed system, and the appendix includes a useful backbone-matched comparison with StreamPETR.

Critical weaknesses
  1. Directly changing the architecture of the LLM might make the pretrained LLM not function. If you want to utilize the pretrained abilities of LLMs for autonomous driving, OnrDrive might completely destroy such abilities. The authors fail to show why VLM is useful and how OneDrive has better insight and makes the VLM-based method better. For example, can the authors prove that changing the structure of the LLM can benefit its text generation ability? Otherwise, VLM is just a large transformer, and its pretrained ability is completely lost.

  2. Insufficient performance improvement. The improvements over existing methods are quite marginal, with no clear evidence of a substantial advantage. The evaluation is also limited to nuScenes open-loop planning and NAVSIM's non-reactive simulation, without real closed-loop testing. This makes it difficult to assess the actual benefits and robustness of the proposed approach.

  3. Insufficient analysis of the proposed architecture. The method combines several existing components, but the paper does not clearly identify which parts are responsible for the improvements. In particular, there is no convincing comparison with a standard cross-attention decoder under matched settings. As a result, it remains unclear whether the proposed unified architecture offers any meaningful advantage over simpler alternatives.

Questions for authors

Please refer to the weakness part for raised questions.

Reviewer_MVQSOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes OneDrive, a unified VLA framework for autonomous driving that combines language generation, perception, and planning within a shared pretrained VLM decoder. It retains pretrained causal attention while introducing task-specific components to support parallel structured prediction. Experiments on nuScenes and NAVSIM demonstrate competitive planning performance and improved inference efficiency.

Critical strengths

1.The paper addresses the mismatch between autoregressive language generation and parallel structured prediction in VLM-based driving systems. Reusing pretrained causal attention while introducing task-specific transformations is a conceptually straightforward approach that may reduce architectural fragmentation and improve backbone reuse. 2.The experiments cover both nuScenes and NAVSIM, and the paper investigates several relevant design choices, including staged training, text supervision, query ordering, decoder depth, and detection queries. The reported results are competitive, and the NAVSIM latency results suggest that the proposed shallow-decoding strategy may offer practical efficiency benefits.

Critical weaknesses

1.The benefit of the proposed shared causal-attention architecture is not sufficiently isolated. The paper lacks a controlled comparison with a conventional parallel query decoder under comparable model capacity and training settings. Consequently, it remains unclear whether the reported gains specifically arise from retaining pretrained causal attention or could be achieved by existing decoder designs. 2.The preservation of language capabilities is not directly validated. The reported text-conditioned trajectory prediction results demonstrate the use of textual inputs for planning, but do not establish whether general language generation or visual question-answering capabilities are retained after driving-oriented fine-tuning. Direct evaluations of these capabilities before and after fine-tuning would provide stronger evidence for this claim.

Questions for authors
  1. Could the authors provide a controlled comparison between OneDrive and a conventional parallel query decoder, using the same visual backbone, training data, training schedule, and comparable model capacity? Such a comparison would help isolate the contribution of retaining pretrained causal attention from that of the additional query self-attention and task-specific FFNs.
  2. Since structured predictions are extracted from shallow mixed layers while deeper LLM layers are retained for language generation, which parts of the decoder are actually shared between planning and language generation? Additionally, could the authors provide direct language-generation or visual-question-answering evaluations before and after driving fine-tuning to substantiate the claim that language capabilities are preserved?
Reviewer_URuuOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents OneDrive, a unified Vision-Language-Action framework for end-to-end driving. Its main idea is to use a pretrained VLM/LLM decoder as a common causal-attention backbone for heterogeneous outputs: parallel structured perception (3D detection and lanes), trajectory planning, and autoregressive text generation. The approach concatenates image, structured query, planning, and text tokens into one sequence. It retains pretrained causal attention, adds query self-attention for perception queries in shallow layers, and replaces language FFNs with task-specific FFNs for detection, lane, and planning tokens. Training proceeds in three stages: perception-language pretraining, planning adaptation, and joint finetuning.

Critical strengths

Reconciling autoregressive language generation with parallel structured prediction inside a VLM-based driving model is an important systems and representation-learning problem. The paper targets a meaningful architectural gap between separate/cascaded decoders and native VLM decoding.

The proposal to place visual tokens before detection, lane, and planning queries lets structured queries condition on visual context using the pretrained causal-attention mechanism. The ordering of detection/lane queries before planning queries also provides a plausible mechanism for planning to access perception-related representations.

The diagnostic finding that pretrained attention transfers more effectively than pretrained FFNs is interesting. Retaining causal attention while using task-specific FFNs is a sensible compromise between reuse of pretrained multimodal relations and task-specific structured prediction.

The staged-training study, text-supervision study, query ordering experiment, planning-loss sweep, and detection-query comparison help establish that several design decisions contribute to the reported performance.

In particular, the authors note the restricted evaluation scope, untested scaling behavior, visual-resolution limitations, and vulnerability to ambiguous/mismatched driving commands

Critical weaknesses
  1. The central framing emphasizes a single pretrained VLM decoder and minimal architectural deviation. However, the method adds structured-query self-attention, separate task-specific FFNs, task heads, 3D positional embeddings, and a multi-stage adaptation protocol. These are reasonable design choices, but they are substantial task-specific components. Thus, the practical contribution is better characterized as a shared-attention multitask decoder with task-specific transformations, rather than a largely unmodified unified VLM decoder.

The paper should explicitly quantify the additional parameters, FLOPs, and training cost introduced by query self-attention and the per-task FFNs. It should also clarify exactly which modules are shared, copied, frozen, LoRA-adapted, or independently learned at each stage.

  1. The diagnostic experiment in Table 1 is suggestive, but not sufficient to support the stronger claim that pretrained VLM causal attention is the key transferable ingredient enabling OneDrive. The study compares initialization choices in a parallel-decoder setting, but the paper would benefit from controlled ablations in the actual OneDrive architecture, such as:

random initialization of causal attention versus pretrained causal attention; frozen pretrained attention versus finetuned attention; replacing causal attention with standard task-specific cross-attention; retaining pretrained FFNs versus task-specific FFNs under matched optimization/training budgets; removing perception-query self-attention; using separate decoders with comparable parameter count and supervision. Without these controls, it is difficult to isolate whether gains arise primarily from pretrained causal attention, the multi-stage curriculum, task-specific FFNs, query ordering, added detection supervision, or their combination.

  1. The nuScenes comparison includes text-based and action-based methods with potentially different inputs, pretraining sources, language annotations, ego-state usage, model sizes, and evaluation conventions. Although the paper distinguishes the two categories, the overall “state-of-the-art” conclusion may not be directly comparable across all rows.

Likewise, NAVSIM comparisons may involve substantially different visual backbones, training data, auxiliary supervision, and sensor modalities. The paper states that OneDrive uses no additional autonomous-driving data for pretraining, but it would be helpful to precisely specify:

all data used for each training stage; whether OmniDrive QA annotations or derived language annotations are used; whether the InternVL3 pretraining data overlap with relevant driving/image domains; model parameter counts for all OneDrive variants; input temporal context, image resolution, camera configuration, and ego-state inputs; the exact NAVSIM training/evaluation protocol and whether all compared methods use identical data splits. Further, all main results should include variation across multiple random seeds or confidence intervals. A difference of 0.05 PDMS or small collision-rate changes can be difficult to interpret without variance estimates.

  1. The evidence for preserved language capability is limited A main claim is that OneDrive preserves multimodal/text generation capability while jointly learning structured driving tasks. However, the language evaluation mainly compares text-conditioned trajectory prediction L2 against OmniDrive. This does not directly evaluate textual generation quality, factual grounding, or whether the language output remains causally useful after joint finetuning.

I encourage the authors to add:

standard driving VQA/text-generation metrics, where applicable; qualitative examples with generated scene descriptions, reasoning, and planned trajectories; human or automatic grounding checks showing whether generated text correctly reflects detected objects, lanes, and ego actions; interventions that alter language instructions while holding images fixed, and measure the resulting behavioral changes; tests for whether text supervision improves behavior under language-dependent or ambiguous scenarios, not only aggregate planning metrics. This is particularly important because Table 5 shows only modest aggregate gains from text supervision.

  1. The 40% latency reduction is compelling for NAVSIM relative to the reported ReCogDrive runtime. However, the nuScenes latency table explicitly notes that attention kernels are not matched: OneDrive uses FlashAttention while several comparison values do not. Therefore, the nuScenes efficiency comparison does not isolate architecture-level efficiency.

Moreover, the complete nuScenes configuration is reported at 513 ms/frame, which raises questions about real-time deployability for a surround-view planning system. The paper should report:

matched-hardware and matched-kernel latency; throughput and memory usage; latency breakdown for ViT encoding, decoder execution, perception heads, and planning; latency as a function of number of cameras, number of image tokens, decoder depth, and output queries; performance–latency trade-offs for the shallow/truncated mode relative to the full model.

  1. The evaluation is restricted to nuScenes and NAVSIM, which the paper itself acknowledges. For a VLA-based framework claiming unified multimodal reasoning and improved planning, stronger tests of generalization would substantially strengthen the paper:

long-tail and safety-critical subsets; adverse weather, nighttime, occlusions, and rare interactions; cross-city or cross-domain transfer; command ambiguity and counterfactual language instructions; out-of-distribution camera conditions; closed-loop scenario-category breakdowns rather than only aggregate PDMS. The limitations section notes ambiguous/mismatched commands, but this should be elevated into a systematic evaluation rather than left primarily as future work.

Questions for authors

In the final model, what percentage of decoder parameters are shared versus task-specific? Please provide parameter counts for the VLM backbone, query self-attention blocks, task-specific FFNs, and heads.

Can the authors provide a controlled ablation that randomizes or freezes the pretrained causal-attention weights within the final OneDrive architecture?

What is the precise source and coverage of the text/QA supervision? Is text available for all training samples, and are the language annotations human-generated, template-generated, or model-generated?

How does the model behave when instructions are changed while visual observations remain fixed? Does the planned trajectory change in semantically appropriate ways?

Are nuScenes results evaluated with the same trajectory horizon, collision protocol, ego-state inputs, map inputs, and standardized evaluation procedure as the compared methods?

Can the authors report mean and standard deviation across multiple seeds for the principal nuScenes and NAVSIM results?

Does the 157 ms NAVSIM configuration use the full model or the truncated shallow-decoder configuration? What is the corresponding PDMS for each inference configuration?

How much does performance degrade in difficult scenarios such as nighttime, occluded intersections, dense traffic, unusual maneuvers, or contradictory commands?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.