- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
MindVLA-U1 uses a shared Transformer to generate language reasoning and continuous driving trajectories. It introduces streaming memory to carry information across frames and intent-based guidance to steer trajectory generation. It also supports planning with or without language generation. Experiments on WOD-E2E report planning accuracy, language quality, inference speed, and component ablations.
Critical strengths
- The paper addresses an important problem: keeping language capabilities while generating precise, continuous trajectories. The shared backbone combines language prediction and action generation, while the memory module uses past information without repeatedly processing full video inputs.
- The method achieves competitive ADE and RFS results on WOD-E2E, with comparisons against several VA and VLA methods. Its fast inference mode achieves VA-level speed. The paper also provides useful implementation details, including model dimensions, memory settings, training schedules, and inference steps, which help readers reproduce the method and experiment.
Critical weaknesses
- The advantage of the shared architecture is not directly demonstrated. The authors argue that passing VLM hidden states to a separate action planner reduces the VLM to a feature extractor and limits language–action coupling. However, they do not compare this design with their shared Transformer while keeping the backbone, data, supervision, memory, and training budget comparable. The reported results therefore do not establish that sharing itself improves planning quality or speed. In particular, skipping language decoding explains the fast/slow speedup, but does not demonstrate an advantage over a separate diffusion planner.
- Evaluation is limited to a single benchmark and open-loop testing. WOD-E2E provides challenging real-world scenes, but generalization to other datasets remains untested. Although the model carries its own memory across frames, the images and vehicle states still come from recorded driving rather than the model’s executed actions. The experiments therefore do not test whether memory and planning remain reliable when prediction errors change later observations. This leaves an important part of the claimed streaming benefits unverified.
Questions for authors
- Could you clarify the benefit of sharing the backbone compared with using the same VLM’s final hidden states to condition a separate diffusion planner? A comparison under similar data, supervision, memory, and training settings would help establish whether sharing improves planning quality or inference speed.
- Could you clarify whether the scores in Figure 1 are based on the authors’ qualitative judgment or supported by experimental results or evidence from prior work? A brief explanation of the scoring criteria and supporting evidence would help readers understand the comparison.
2 comments
Closed-loop streaming evaluation on real-world in-house benchmarks
4. Visual closed-loop evaluation of streaming memory
To directly address Reviewer 3 (PCw4)'s concern about streaming under action-dependent observations, we conduct additional visual closed-loop experiments on an internal driving benchmark distinct from WOD-E2E.
Training setup. We use a separately trained MindVLA-U1 checkpoint. SFT uses approximately 1.85 million driving clips containing 11.6 million anchor frames. Starting from the SFT checkpoint at step 100,000, we apply GRPO to update the trajectory generator on approximately 2,000 reconstructed driving scenes. We evaluate the checkpoint after one epoch of RL updates.
Visual closed-loop protocol. During RL, the policy replans at 1 Hz using rendered multi-view images and simulated ego states. A tracking controller executes the planned trajectories, while the simulator updates vehicle dynamics at 10 Hz. The updated ego pose determines subsequent rendered camera observations, so planning errors affect later observations and decisions. Surrounding traffic agents follow their logged trajectories.
Paired memory ablation. We evaluate an internal suite of 256 scenarios covering the evaluation dimensions. Each scenario is evaluated using the same checkpoint with streaming visual memory enabled and disabled, keeping the ego-history window and navigation settings identical. “Non-streaming” and “Streaming” below refer to memory disabled and enabled at inference, respectively.
| Dimension | Non-streaming | Streaming | Difference |
|---|---|---|---|
| Safety | 38.92 | 40.80 | 1.87 |
| Comfort | 82.21 | 82.60 | 0.39 |
| Efficiency | 70.30 | 71.89 | 1.59 |
| Compliance | 77.27 | 78.00 | 0.74 |
| Total Score | 67.17 | 68.32 | 1.15 |
The total score is the arithmetic mean of Safety, Comfort, Efficiency, and Compliance. Enabling streaming memory improves the total score from 67.17 to 68.32 (+1.15), with gains across all four dimensions. This directly supports the benefit of streaming memory under visual closed-loop feedback, where the policy's executed actions change subsequent observations rather than merely consuming a fixed sequence of logged inputs.
Shared-backbone ablations, additional NAVSIM results and clarification for Fig. 1
1. Benefits of the shared backbone
To ensure fair comparisons, we keep the input, context order, training data, supervision, training budget, memory, dimensions, diffusion settings, and evaluation protocol identical across all variants. The 5 variants differ only in architectural interfaces and gradient:
We have the following interpretations:
2. Reactive NAVSIM benchmarks with streaming inference
We report closed-loop results on NAVSIM v1, v2, and Navhard (all TOP-1) to showcase the cross-benchmark applicability of the U1 design under official evaluation protocols. All U1 models initialize from pretrained VLM weights, without additional driving-data pretraining, and adopt the same streaming behavior for training and inference.
As shown in the table, U1 achieves the highest TOP-1 scores among the listed baselines across all three benchmarks.
3. Clarification for Figure 1
Figure 1 combines reported quantitative evidence (from official papers) with qualitative assessments accessed by ourselves. For example, U1's flexible-compute capability is supported by its fast/slow modes. RAP-DINO is rated low on language preservation because its compared VA configuration does not provide a language interface.
The radial positions are illustrative qualitative assessments, not measured, normalized scores. The figure's main purpose is to emphasize that practical L4 autonomous driving is a complex, multidimensional problem in which multiple constraints must be addressed jointly. Excellence on an isolated dimension is insufficient for real-world deployment; the figure motivates developing these capabilities together as a whole.
All clarifications and corresponding revisions discussed above will be incorporated into the final version.