acceptodds
Under review as a conference paper at ICLR 2027

MindVLA-U1: VLA Beats VA with Unified Streaming Architecture for Autonomous Driving

Abstract

Autonomous driving has progressed from modular pipelines toward end-to-end unification, and Vision-Language-Action (VLA) models are a natural extension of this journey beyond Vision-to-Action (VA). In practice, driving VLAs have often trailed VA on planning quality, suggesting that the difficulty lies in the interface through which semantic reasoning, temporal context, and continuous control are combined. We argue that this gap reflects how VLA has been built—as isolated subtask improvements that fail to compose into coherent driving capabilities—rather than what VLA is. We present MindVLA-U1, the first unified streaming VLA architecture for autonomous driving. A unified VLM backbone produces autoregressive (AR) language tokens (optional) and flow-matching (diffusion-style) continuous action trajectories in a single forward pass over one shared representation, preserving the natural output form of each modality. A full streaming design processes the driving video framewise rather than as fixed video-action chunks under costly temporal VLM modeling. Planned trajectories evolve smoothly across frames while a learned streaming memory channel carries temporal context and updates. The unified architecture enables fast/slow systems on dense/sparse Mixture-of-Transformers (MoT) backbones via flexible self-attention context management, and exposes a measurable language-control path for action: a language-predicted driving intent steers the action diffusion via classifier-free guidance (CFG), turning language-side intent into control signals for continuous action planning. On the long-tail WOD-E2E benchmark, MindVLA-U1 surpasses experienced human drivers for the first time (8.20 RFS vs. 8.13 GT RFS) with only 2 diffusion steps, achieves state-of-the-art planning ADEs over prior VA/VLA methods by large margins, and matches VA-class latency (∼16 FPS vs. RAP's ∼18 FPS at matched ∼1B scale) while preserving natural language interfaces for human–vehicle interaction.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-Language-Action models, Autonomous Driving

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_PCw4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MindVLA-U1 uses a shared Transformer to generate language reasoning and continuous driving trajectories. It introduces streaming memory to carry information across frames and intent-based guidance to steer trajectory generation. It also supports planning with or without language generation. Experiments on WOD-E2E report planning accuracy, language quality, inference speed, and component ablations.

Critical strengths
  1. The paper addresses an important problem: keeping language capabilities while generating precise, continuous trajectories. The shared backbone combines language prediction and action generation, while the memory module uses past information without repeatedly processing full video inputs.
  2. The method achieves competitive ADE and RFS results on WOD-E2E, with comparisons against several VA and VLA methods. Its fast inference mode achieves VA-level speed. The paper also provides useful implementation details, including model dimensions, memory settings, training schedules, and inference steps, which help readers reproduce the method and experiment.
Critical weaknesses
  1. The advantage of the shared architecture is not directly demonstrated. The authors argue that passing VLM hidden states to a separate action planner reduces the VLM to a feature extractor and limits language–action coupling. However, they do not compare this design with their shared Transformer while keeping the backbone, data, supervision, memory, and training budget comparable. The reported results therefore do not establish that sharing itself improves planning quality or speed. In particular, skipping language decoding explains the fast/slow speedup, but does not demonstrate an advantage over a separate diffusion planner.
  2. Evaluation is limited to a single benchmark and open-loop testing. WOD-E2E provides challenging real-world scenes, but generalization to other datasets remains untested. Although the model carries its own memory across frames, the images and vehicle states still come from recorded driving rather than the model’s executed actions. The experiments therefore do not test whether memory and planning remain reliable when prediction errors change later observations. This leaves an important part of the claimed streaming benefits unverified.
Questions for authors
  1. Could you clarify the benefit of sharing the backbone compared with using the same VLM’s final hidden states to condition a separate diffusion planner? A comparison under similar data, supervision, memory, and training settings would help establish whether sharing improves planning quality or inference speed.
  2. Could you clarify whether the scores in Figure 1 are based on the authors’ qualitative judgment or supported by experimental results or evidence from prior work? A brief explanation of the scoring criteria and supporting evidence would help readers understand the comparison.
2 comments
AuthorsOpenReview ↗

Shared-backbone ablations, additional NAVSIM results and clarification for Fig. 1

1. Benefits of the shared backbone

To ensure fair comparisons, we keep the input, context order, training data, supervision, training budget, memory, dimensions, diffusion settings, and evaluation protocol identical across all variants. The 5 variants differ only in architectural interfaces and gradient:

  1. Standard U1: Context and action tokens are processed within one shared 2B backbone.
  2. Action-to-context K/V stop-gradient: We detach the context K/V caches used by action attention. This blocks gradients through that path while preserving forward context access.
  3. Separate 2B expert, detached features: The original 2B VLM aggregates context, while a separate 2B action expert performs diffusion conditioned on the context VLM's detached outputs, without sharing its layer-wise K/V caches.
  4. Separate 2B expert, end-to-end gradients: Uses the same architecture as row 3, but allowing action-loss gradients to propagate into the context VLM backbone.
  5. Lightweight MLP head, detached features: Replaces the separate 2B expert with the same lightweight MLP action-readout used in U1, conditioned on detached VLM outputs.
VariantBLEU-4 / ROUGE-L ↑RFS-GT ADE 3/5s (m) ↓RFS-matched ADE 3/5s (m) ↓RFS ↑
MindVLA-U1 (no intent)0.30 / 0.490.92 / 2.140.50 / 1.057.83
Action-to-context K/V stop-gradient0.30 / 0.480.94 / 2.180.51 / 1.087.72
Separate 2B action expert, detached final VLM features0.29 / 0.491.01 / 2.260.56 / 1.147.54
Separate 2B action expert, end-to-end gradients0.23 / 0.440.97 / 2.210.53 / 1.107.66
Lightweight action head, detached final VLM features0.29 / 0.481.08 / 2.400.60 / 1.227.34

We have the following interpretations:

  • Context gradients matter (rows 1 vs. 2): Stopping gradients reduces planning quality despite preserving forward context access, supporting the benefit of action-supervised context learning.
  • Restoring gradients alone is insufficient (rows 3 vs. 4, then 4 vs. 1): Restoring gradients improves planning but reduces language quality in the separate-expert architecture. It still trails U1 in both, suggesting that gradient connectivity alone does not recover the integrated model's joint capabilities.
  • A thin readout is not a thin planner (rows 3 vs. 5): The separate 2B expert outperforms the lightweight head under the detached final-feature interface. U1's thin head, however, reads action features already processed by the full shared backbone; applying that head directly to final context features does not reproduce this computation.

2. Reactive NAVSIM benchmarks with streaming inference

We report closed-loop results on NAVSIM v1, v2, and Navhard (all TOP-1) to showcase the cross-benchmark applicability of the U1 design under official evaluation protocols. All U1 models initialize from pretrained VLM weights, without additional driving-data pretraining, and adopt the same streaming behavior for training and inference.

Methodv1 PDMS ↑v2 EPDMS ↑Navhard EPDMS ↑First releaseVenue / status
MindVLA-U195.292.257.1——
ChainFlow-VLA94.8——2026-05-22NeurIPS 2026
TOAD94.7—56.32026-06-05arXiv
DrivoR-Scale94.6—54.62026-01-08CVPR 2026
CLOVER94.590.448.32026-05-14arXiv
RAP-DINO93.8—36.92025-10-05ICLR 2026
DriveSuprim93.587.1—2025-06-07AAAI 2026
DrivoR93.1—48.32026-01-08CVPR 2026
SparseDriveV292.090.1—2026-03-31ECCV 2026
DriveFine91.889.7—2026-02-16ECCV 2026
ReCogDrive90.883.6—2025-06-09ICLR 2026
MeanFuser89.089.5—2026-02-23CVPR 2026
GuideFlow——51.52025-11-24CVPR 2026
SimScale—85.953.32025-11-28CVPR 2026

As shown in the table, U1 achieves the highest TOP-1 scores among the listed baselines across all three benchmarks.


3. Clarification for Figure 1

Figure 1 combines reported quantitative evidence (from official papers) with qualitative assessments accessed by ourselves. For example, U1's flexible-compute capability is supported by its fast/slow modes. RAP-DINO is rated low on language preservation because its compared VA configuration does not provide a language interface.

The radial positions are illustrative qualitative assessments, not measured, normalized scores. The figure's main purpose is to emphasize that practical L4 autonomous driving is a complex, multidimensional problem in which multiple constraints must be addressed jointly. Excellence on an isolated dimension is insufficient for real-world deployment; the figure motivates developing these capabilities together as a whole.


All clarifications and corresponding revisions discussed above will be incorporated into the final version.

AuthorsOpenReview ↗

Closed-loop streaming evaluation on real-world in-house benchmarks

4. Visual closed-loop evaluation of streaming memory

To directly address Reviewer 3 (PCw4)'s concern about streaming under action-dependent observations, we conduct additional visual closed-loop experiments on an internal driving benchmark distinct from WOD-E2E.

Training setup. We use a separately trained MindVLA-U1 checkpoint. SFT uses approximately 1.85 million driving clips containing 11.6 million anchor frames. Starting from the SFT checkpoint at step 100,000, we apply GRPO to update the trajectory generator on approximately 2,000 reconstructed driving scenes. We evaluate the checkpoint after one epoch of RL updates.

Visual closed-loop protocol. During RL, the policy replans at 1 Hz using rendered multi-view images and simulated ego states. A tracking controller executes the planned trajectories, while the simulator updates vehicle dynamics at 10 Hz. The updated ego pose determines subsequent rendered camera observations, so planning errors affect later observations and decisions. Surrounding traffic agents follow their logged trajectories.

Paired memory ablation. We evaluate an internal suite of 256 scenarios covering the evaluation dimensions. Each scenario is evaluated using the same checkpoint with streaming visual memory enabled and disabled, keeping the ego-history window and navigation settings identical. “Non-streaming” and “Streaming” below refer to memory disabled and enabled at inference, respectively.

DimensionNon-streamingStreamingDifference
Safety38.9240.801.87
Comfort82.2182.600.39
Efficiency70.3071.891.59
Compliance77.2778.000.74
Total Score67.1768.321.15

The total score is the arithmetic mean of Safety, Comfort, Efficiency, and Compliance. Enabling streaming memory improves the total score from 67.17 to 68.32 (+1.15), with gains across all four dimensions. This directly supports the benefit of streaming memory under visual closed-loop feedback, where the policy's executed actions change subsequent observations rather than merely consuming a fixed sequence of logged inputs.

Reviewer_rJnqOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes MindVLA-U1, a driving VLA built on Qwen3-VL-2B. Vision, ego-state, memory, optional language and noisy trajectory tokens share the same transformer weights; an LM head decodes language autoregressively, and a 2-layer MLP predicts a flow-matching velocity for a 5 s, 20-waypoint trajectory (2 Euler steps at inference). Three components are added: (1) a streaming memory, in which a Q-Former compresses each step's backbone outputs into 128 tokens kept in a 2-entry FIFO and ego-motion-modulated when read; (2) Intent-CFG, in which the LM head predicts WOD-E2E's 3-class intent (left/right/straight) and the predicted token conditions the action head through classifier-free guidance; (3) inference modes defined by attention masks (language-first, action-first, action-only), plus a two-group Mixture-of-Transformers variant for a fast path. Language supervision comes from MindLabel, VQA pairs auto-generated by Qwen3-VL and Qwen3.5-Plus (~1.95M on the training split). All experiments are open-loop on WOD-E2E. Reported results: val RFS 7.83 (dense), 7.92 (+Intent-CFG or +MoT), and 8.20 after GRPO with RFS as the reward, above the 8.13 of the logged trajectory; official test RFS 7.77, and 7.87 after RL with RFS-GT ADE 1.09/2.66 m; latency of 108 ms for the 2B fast path and 64 ms for a 1B InternVL-2 variant, against 57 ms for RAP-DINO. Ablations cover intent sources, MoT grouping, streaming and memory, and a 4-frame VLM input.

Critical strengths
  1. A practically useful integration of continuous planning and optional language generation. The architecture preserves continuous trajectory outputs while supporting multiple language/action execution modes within one checkpoint. Table 5 reports similar planning performance with autoregressive answer decoding enabled or bypassed (7.83 versus 7.82 RFS). This supports the practical value of flexible execution modes, although it does not by itself establish that generated language improves planning.
  2. Useful experimental evidence for the streaming-memory component. Table 6 separates streaming training from the memory channel. Adding memory to the streaming baseline improves the reported RFS from 7.73 to 7.83 and reduces 5-second rater-matched ADE from 1.14 to 1.05 m. This provides relevant component-level evidence for temporal-state propagation. The competitive non-RL test result also indicates that the system’s planning performance is not solely attributable to RL post-training.
Critical weaknesses
  1. The strongest performance claims exceed the evidence presented. Table 2 omits stronger published official-test results from already-cited work: RAP-DINO reports 8.04 RFS and Poutine reports 7.99, compared with 7.87 for MindVLA-U1 + RL. RAP-DINO also reports 5-second ADE of 2.65 m versus 2.66 m here. Assuming comparable evaluation protocols, these results do not support the unqualified claims of the highest official-test RFS or uniformly superior planning ADEs. The proposed system remains competitive, particularly in its reported 3-second ADE and use of no external driving-data pretraining.
  2. The key intent-conditioning experiment does not isolate the source of the planning improvement. In Table 3, the NTP-predicted variant improves RFS from 7.83 to 7.92, but it is also paired with prototype-grounded embedding refinement. The manuscript does not establish that the alternative intent sources use an equivalent refinement setup, and Appendix D does not define the refinement sufficiently to reproduce it. Thus, despite the statement that the comparison isolates the conditioning-signal source, the improvement could arise from the intent source, the embedding refinement, or their interaction. This matters because the paper uses this experiment to support its central language-to-action argument. The results demonstrate intent-conditioned trajectory control, but do not yet establish which part of the proposed mechanism accounts for the aggregate planning gain.
Questions for authors
  1. RL evaluation: What data were used for RL reward optimization, checkpoint selection, and final evaluation? Was the reported 8.20 RFS obtained on samples held out from both optimization and selection, and was the 8.13 logged-reference score computed on those same samples?
  2. Intent-conditioning attribution: Is prototype-grounded embedding refinement exclusive to the NTP variant in Table 3? Please define the refinement and clarify whether the variants otherwise share the same conditioning and training setup. Is there a comparison that holds refinement fixed while varying the intent source?
  3. Benchmark comparability: Why are the published RAP-DINO and Poutine test results absent from Table 2? Please clarify any differences in evaluation protocol or ADE reference trajectories, and explain how including comparable results affects the claimed official-test leadership.
2 comments
AuthorsOpenReview ↗

Benchmark Comparisons, Intent Conditioning, and RL Evaluation Protocol

1. Benchmark comparisons and claim scope

ADE and RFS are official-test results. Language scores are evaluated by us on MindLabel-annotated VQA using the same protocol for the SFT/RL checkpoints. The full comparison below shows mixed results on the WOD-E2E test set:

MethodBLEU-4 ↑ROUGE-L ↑RFS-GT ADE 3s (m) ↓RFS-GT ADE 5s (m) ↓RFS ↑
RAP-DINO——1.172.658.04
Poutine——1.212.747.99
MindVLA-U1, SFT0.310.511.162.677.77
MindVLA-U1, RL0.310.501.092.657.87

We agree that Table 2 should include RAP-DINO and Poutine and that U1 does not achieve the highest official-test RFS. The expanded comparison instead shows mixed results:

  • Both U1 variants achieve lower 3-second ADE than RAP-DINO.
  • Conversely, RAP-DINO and Poutine achieve higher RFS than U1-RL.

For a controlled comparison in Table 1, we retrain RAP-DINO on the same WOD-E2E data as U1 without extra driving data, and evaluate both on the same held-out validation split under matched settings . We will revise the claims to reflect these metric-dependent strengths rather than uniform superiority.

We additionally compare BLEU-4 and ROUGE-L before and after RL post-training, showing consistent planning gains while preserving language performance.

Note that RFS reflects human-rater preferences rather than an interpretable geometric error, and the benchmark provides limited visibility into the RFS scoring rationale. Our scenario statistics (Appendix Table 8) also indicate a substantial train/validation-to-test distribution shift: awaiting scenarios are rare in training/validation but considerably more frequent in the test set. We leave studying whether broader pretraining improves robustness to these OOD scenarios to future work.

We also compare U1 with RAP-DINO on NAVSIM in our response to Reviewer 3 PCw4. U1 achieves 95.2 PDMS on NAVSIM v1, compared with RAP-DINO's 93.8, alongside 92.2 EPDMS on NAVSIM v2 and 57.1 EPDMS on Navhard, achieving the highest TOP-1 scores among the compared baselines across all three benchmarks. These explicitly scored benchmarks provide complementary evidence beyond WOD-E2E and RFS.

2. Intent-conditioning clarification and attribution

We apologize for the misleading phrase “prototype-grounded embedding refinement.” No additional prototype-construction or embedding-refinement module is involved. Both variants use the same three GT-defined intent classes, token vocabulary, embeddings, and action-conditioning pathway. The only difference is whether the intent token is externally specified by the recorded GT label or inferred from scene context through NTP. By “refinement,” we intended to describe scene-dependent intent selection informed by the LLM's learned semantic priors, not an additional embedding transformation. All other experimental settings—including the backbone, data, supervision, memory, training budget, and diffusion/CFG settings—are identical.

WOD-E2E is specifically designed to evaluate driving decisions and trajectory planning in challenging cornercase scenarios. In these situations, GT intent labels the recorded human maneuver, which may reflect an immediate reaction under urgent conditions. It describes the behavior that occurred, but does not exhaust the reasonable driving choices available in the scene. NTP infers intent through the LLM's pretrained language-prediction pathway, allowing its learned semantic knowledge to inform the driving decision. In contrast, the standalone MLP learns a task-specific three-way classification head from the same intent supervision. Under matched settings, this classifier achieves 7.76 RFS, compared with 7.92 RFS for NTP-based prediction (0.16 RFS lower). NTP outperforms GT-intent conditioning, whereas the classifier underperforms it. This ordering shows that predicting intent alone is insufficient and supports leveraging the LLM's NTP pathway for scene-dependent driving decisions.

Figure 5(b) provides complementary qualitative evidence of richer language-conditioned control. Moving from the GT 3-class intents to 20 MindLabel-annotated intents extends the conditioning vocabulary beyond left/right/straight to behaviors such as U-turns. For example, 07 (U-turn) produces trajectory patterns more similar to 05 (turn left) than to 06 (turn right), consistent with the US right-hand-traffic setting of WOD-E2E. This semantic consistency illustrates the potential for leveraging the LLM's semantic and compositional knowledge for richer driving decisions, motivating further exploration of zero-shot decision-making and driving-behavior control.

We will correct the terminology and clarify the conditioning interface and comparison settings.


3. RL optimization, selection, and evaluation

RL reward optimization and checkpoint selection use 128 randomly sampled validation samples out of 479. The remaining 351 are held out from both and used for evaluation. The reported 8.20 RFS is obtained under this 128/351 protocol, not by evaluating on the 128 optimization samples.

We also tested multiple random choices of the 128 optimization samples, evaluating each resulting model on the remaining 351 samples, and observed consistent held-out RFS from 8.19 to 8.22. For clarity, using all 479 validation samples for both RL optimization and evaluation can yield RFS 8.68. This is an in-sample result, not held-out performance, and is distinct from the reported 8.20 evaluation.

The previously cited GT RFS of 8.13 comes from the Poutine paper. On the same 351 held-out samples used for our 8.20 RFS evaluation, the GT trajectory reference obtains 8.12 RFS. We will clarify this in the final version.

Reviewer_WhcCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MindVLA-U1 combines autoregressive language modeling and continuous flow-matching trajectory prediction in a shared transformer backbone. A compact streaming memory carries information across frames, while configurable attention masks support different language/action execution orders. The paper also introduces intent-conditioned classifier-free guidance and a Mixture-of-Transformers variant. Training combines driving trajectories with generated question–answer supervision, followed optionally by RFS-based reinforcement learning. On WOD-E2E, the model reports strong official-test trajectory accuracy and RFS, with further RFS gains after reinforcement learning. The experiments study intent conditioning, streaming memory, expert routing, and inference speed. The main contribution is an architecture that supports both language outputs and continuous planning, with several operating points trading computation for capability.

Critical strengths
  1. The shared architecture produces useful results before RL. The non-RL model reaches 7.77 RFS and 1.16/2.67 m ADE on the official test split in Table 2, so the result is not solely an effect of reward optimization. Table 6 also separates streaming training from adding memory, rather than presenting them as one inseparable improvement. Supporting language and continuous actions within the same model is appealing, and the timing breakdown makes the cost of language decoding unusually clear.
Critical weaknesses
  1. The intent experiment changes more than the intent source. Table 3 gives the predicted-intent variant a prototype-grounded embedding refinement that the other variants apparently do not receive. Its RFS improvement from 7.83 to 7.92 therefore cannot be attributed cleanly to language-predicted intent. Appendix D repeats that this refinement is used but does not specify how the prototypes are constructed or optimized. A matched embedding comparison is especially important because the oracle-intent variant shows no aggregate improvement.

  2. The headline performance and speed come from different configurations. The 8.20 validation RFS uses RL, whereas the roughly 16 FPS result uses an InternVL-2 1B configuration with template QA and 7.78 RFS. The main 2B model takes 108 ms with template QA and 2,594 ms with decoded answers; Table 5 also excludes intent conditioning. These are useful operating points, but together they do not demonstrate that one configuration delivers the claimed planning quality, active language reasoning, and VA-like throughput simultaneously.

Questions for authors
  1. Could you fully specify the prototype refinement and run a factorial ablation crossing intent source with embedding design? Please keep training and inference settings matched and include a learned three-class classifier outside the language head to test what language prediction specifically contributes.

  2. Could you report RFS, ADE, VQA quality, and end-to-end latency for the same checkpoints with RL, Intent-CFG, and actual answer decoding enabled or disabled? Please include uncertainty for the 8.20 versus 8.13 RFS comparison and frame it specifically as an open-loop metric result.

2 comments
AuthorsOpenReview ↗

Intent Conditioning, Configuration-Matched Evaluation, RFS Uncertainty, and more Closed-loop Results

1. Intent-conditioning clarification and classifier comparison

We apologize for the misleading phrase “prototype-grounded embedding refinement.” No additional prototype-construction or embedding-refinement module is involved. Both variants use the same three GT-defined intent classes, token vocabulary, embeddings, and action-conditioning pathway. The only difference is whether the intent token is externally specified by the recorded GT label or inferred from scene context through NTP. By “refinement,” we intended to describe scene-dependent intent selection informed by the LLM's learned semantic priors, not an additional embedding transformation. All other settings are matched in Table 3. We will correct the terminology and clarify these settings.

WOD-E2E is specifically designed to evaluate driving decisions and trajectory planning in challenging cornercase scenarios. In these situations, GT intent labels the recorded human maneuver, which may reflect an immediate reaction under urgent conditions. It describes the behavior that occurred, but does not exhaust the reasonable driving choices available in the scene. NTP infers intent through the LLM's pretrained language-prediction pathway, allowing its learned semantic knowledge to inform the driving decision. In contrast, the standalone MLP learns a task-specific three-way classification head from the same intent supervision. As also reported in our response to Reviewer 2 rJnq, this classifier achieves 7.76 RFS under matched settings, compared with 7.92 RFS for NTP-based prediction. NTP outperforms GT-intent conditioning, whereas the classifier underperforms it. This ordering shows that predicting intent alone is insufficient and supports leveraging the LLM's NTP pathway for scene-dependent driving decisions.

Figure 5(b) provides complementary qualitative evidence. Extending the GT 3-class vocabulary to 20 MindLabel-annotated intents supports richer behavior conditioning: 07 (U-turn) produces trajectory patterns closer to 05 (turn left) than to 06 (turn right), consistent with WOD-E2E's right-hand-traffic setting. This motivates further exploration of LLM-based compositional and zero-shot driving control.

2. Inference modes and configuration-specific results

Table 5 reports the SFT model's latency with actual QA answer decoding enabled or bypassed. To address the requested RL + Intent-CFG configurations, we extend this same model while preserving its fast/slow execution modes. RL updates the model weights without introducing additional inference modules or forward passes. During RL post-training, U1 uses Intent-CFG to generate multiple plausible trajectory proposals for reward-based optimization. Intent-CFG is therefore enabled by default in the RL configuration.

Intent conditioning introduces a single predicted intent token. Before distillation, CFG evaluates both conditional and unconditional predictions. After RL post-training, we apply CFG distillation to combine these into a single student forward pass per diffusion step, following cfg-distillation techniques used in video generation. The resulting deployment model therefore retains the original execution path with one additional intent token, while full answer decoding remains independently switchable.

The following table reports configuration-matched measurements for the SFT and distilled RL checkpoints, with QA decoding toggled within each checkpoint:

ConfigurationQA decodingRFS ↑RFS-matched ADE 3/5s (m) ↓BLEU-4 / ROUGE-L ↑Latency (ms) ↓
Qwen3-VL-2B, SFT, no intentOff7.820.51 / 1.05—108
Same SFT checkpointOn7.830.50 / 1.050.30 / 0.492,594
Qwen3-VL-2B, RL + Intent-CFG, distilledOff8.190.51 / 1.06—111
Same distilled checkpointOn8.190.50 / 1.040.31 / 0.492,617

Measurements use batch size 1 on one H200, matching the settings in Table 5. “—” means no QA answer is generated in that execution mode.

We will explicitly distinguish these operating points. With QA decoding bypassed, the 2B model retains approximately 9 FPS after RL and CFG distillation (111 ms versus 108 ms for the SFT baseline), showing negligible latency overhead from the additional intent token. Our claim is selectable quality–compute operating points, rather than simultaneous peak RFS, full answer generation, and peak throughput. We will clarify this in the final version.


3. Held-out RFS and uncertainty

RFS annotations are available only for the 479 validation samples; we randomly select 128 for RL reward optimization and checkpoint selection, reserving the remaining 351 exclusively for evaluation.

On the reported held-out split, the original RL checkpoint achieves 8.20 RFS versus 8.12 for the GT trajectory reference on the same samples, while the CFG-distilled checkpoint achieves 8.19.

Across 6 random 128/351 splits, held-out RFS is 8.20 ± 0.015, with an observed range of 8.19–8.22, and the paired model-minus-GT difference is 0.08 ± 0.01, using each split's own GT reference. Here, ± denotes sample standard deviation across splits, not a confidence interval.

The previously cited 8.13 came from Poutine paper rather than this held-out subset. We will replace it with the matched-sample reference in the final paper.

We will explicitly frame the WOD-E2E RFS comparisons as open-loop metric results, not evidence of human-level or superior closed-loop driving. For additional benchmark evaluations, please refer to our response to Reviewer 3 PCw4. U1 achieves the highest TOP-1 scores among the compared baselines across NAVSIM v1, NAVSIM v2, and Navhard, using the same streaming architecture without additional driving-data pre-training.

Comments on the paper

1 comment
AuthorsOpenReview ↗

To ACs and all reviewers: Overall Clarification of the Contribution and Summary of Additional Evidence

We would like to clarify our central contribution and its broader significance, and present additional results for U1 on more benchmarks, including NAVSIM and our in-house closed-loop RL evaluations with real-scene visual rollouts:

  1. Architectural unification for coupled real-world L4 driving constraints. U1's core novelty is a shared streaming architecture that unifies autoregressive language and continuous flow-matching actions while preserving their distinct output forms. Reconciling their different attention, generation, and temporal requirements is a non-trivial joint-design problem, not simply combining output heads or independently optimized modules. Intent-CFG adds explicit language-to-action alignment, using predicted semantic intent to guide continuous trajectory generation. Flexible MoT extensions, fast/slow execution, and rich MindLabel supervision further support the joint requirements of planning quality, temporal consistency, semantic interaction, and computational efficiency.

  2. Consistent performance across benchmarks. In response to Reviewer 2 (rJnq)'s benchmark-comparison concerns, we provide additional results for U1 on NAVSIM v1, NAVSIM v2, and Navhard, achieving the highest TOP-1 scores among the compared methods: 95.2 PDMS, 92.2 EPDMS, and 57.1 EPDMS, respectively. These results support the architecture's effectiveness and broad applicability, rather than a single-benchmark or metric-specific improvement.

  3. Effective on-policy RL with real-scene visual rollouts. In response to Reviewer 3 (PCw4)'s concern about closed-loop streaming, we report on-policy RL results using action-dependent visual rollouts on our in-house real-world benchmarks. Open-source, reconstruction-based visual closed-loop frameworks (e.g., AlpaSim with 3DGS scene-wise reconstructions) can suffer from rapid and severe visual degradation and temporal inconsistency as policy rollouts depart from recorded camera viewpoints. The resulting artifacts and sim-to-real mismatch can confound both gains and failures, making it unclear whether performance reflects driving decisions or rendering quality. Our in-house platform maintains stable visual feedback over long driving horizons, reducing this source of confounding. The results quantify the closed-loop benefits of streaming memory after on-policy RL training, under policy-induced changes in camera observations and ego states.


We will incorporate all clarifications, corrections, additional results, and corresponding revisions discussed in our responses to all reviewers into the final version. Code will be released upon acceptance for reproduction / reference.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.