acceptodds
Under review as a conference paper at ICLR 2027

World-Language-Action Models for Autonomous Driving

Abstract

Autonomous driving requires understanding the observed scene, predicting the consequences of actions, and planning safe trajectories. Existing efforts to combine these capabilities often use separate visual representations for understanding and generation. We propose Doe-3, a world-language-action model that jointly models world states, language, and actions. We adopt the pretrained driving world representation to encode synchronized multi-view observations into scene tokens that retain appearance, semantics, and geometry. We introduce a shared autoregressive backbone that treats this world representation as a modality alongside language and action. The backbone provides a common temporal context for grounded language responses, trajectory planning, and action-conditioned future prediction. We attach a lightweight diffusion head that maps autoregressive world tokens to future scene latents, which pretrained decoders convert into RGB, depth, semantic, and occupancy outputs. We jointly train the backbone with language, planning, and future-prediction objectives to connect semantic understanding with scene dynamics. We adopt a three-stage schedule that progressively aligns the scene interface, adapts the backbone, and jointly optimizes the model. Doe-3 achieves a PDMS of 94.0 on NAVSIMv1 and an EPDMS of 90.8 on NAVSIMv2, outperforming the strongest prior methods. On nuScenes, it reduces the future-depth absolute relative error to 0.13 and reaches 27.03 mIoU and 61.49 IoU in camera-based 4D occupancy forecasting.

Primary area:
applications to robotics, autonomy, planning
Keywords:
End-to-End Autonomous Driving, Driving World Model, Vision-Language Model, Unified Generation and Understanding

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_zi2KOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Doe-3 is a unified world-language-action model. Its core method encodes synchronized multi-view observations into scene tokens that preserve appearance, semantics and geometry via a pre-trained driving world representation. It unifies these tokens with language and action modalities through a shared autoregressive backbone, and equips a lightweight diffusion head to map world tokens to multi-modal future scene outputs. The model is progressively aligned and optimized in three stages, realizing deep coupling between semantic understanding and scene dynamics.

Critical strengths

The framework proposed by the authors is relatively concise, and it effectively unifies multiple modalities. Its core advantages are as follows:

  1. ‌Unified multi-modal representation‌ eliminates redundant independent visual encoders, integrating multi-view scene tokens, language and driving actions into one shared autoregressive backbone, breaking long-existing representation barriers across perception, prediction and planning.
  2. Natively coupled multi-task learning‌ connects semantic understanding directly with scene dynamics via joint language, planning and future prediction objectives, supporting grounded language response, safe trajectory planning and action-conditioned future forecasting in a single forward pass.
Critical weaknesses
  1. The proposed method is relatively concise, but there are many difficulties in training such a model. The authors proposed a three-stage training framework, but I haven't seen more details about these three stages of training, such as what kind of data is used in each stage. As the objectives of each stage are different, I think that the data and the training strategies used should also differ.
  2. From the proposed framework, all image tokens, text tokens, and other tokens are intertwined into a unified sequence, where each token can only access itself and all tokens preceding it. However, for the tokens of a single image, they can see each other. Will this paradigm have a negative impact on image generation?
Questions for authors

Please see weaknesses above.

1 comment
AuthorsOpenReview ↗

Author Response

We sincerely thank the reviewer for the encouraging remarks and constructive feedback. Below, we provide the clarified details of the three-stage training schedule and explain how our attention design simultaneously ensures temporal causality and full spatial interaction.


W1 — Three-Stage Training: Data, Modules, and Objectives

Thank you for pointing out the inconsistency. Appendix A inadvertently omitted the language objective in the final stage, which will be corrected. The clarified training schedule is summarized in Table 1.

Table 1: Three-stage progressive optimization schedule. P, G, and L denote planning, future world generation, and language objectives, respectively.

StageTrainable ModulesObjectivesλL\lambda_LOpenScene / nuScenes EpochsLearning Rate (LR)
1: Interface AlignmentScene interfaces, embeddings, and query tokensP+G01 / 401e-4
2: Backbone AdaptationShared backbone LoRAP+G+L0.102 / 405e-5
3: Joint OptimizationLoRA, interfaces, planning head, and designated diffusion conditioning modulesP+G+L0.055 / 805e-5

In Stage 1, the backbone and downstream task heads are frozen, optimizing only the scene interface via planning and generative world-modeling losses. Stage 2 adapts the shared backbone via LoRA while introducing language supervision. Stage 3 jointly fine-tunes the interfaces, shared LoRA adapters, and designated task modules while retaining all three training objectives. All stages utilize both OpenScene and nuScenes for the epochs specified in Table 1, computing supervisory losses according to available task annotations.

The composite loss function is defined as:

L=3Llatent+4Lflow+0.5Ltraj+Ldyn+0.75Lscore+λLLCE\mathcal{L} = 3\mathcal{L}_{\mathrm{latent}} + 4\mathcal{L}_{\mathrm{flow}} + 0.5\mathcal{L}_{\mathrm{traj}} + \mathcal{L}_{\mathrm{dyn}} + 0.75\mathcal{L}_{\mathrm{score}} + \lambda_L\mathcal{L}_{\mathrm{CE}}

The terms represent, respectively: future latent MSE, flow matching on latent velocity fields, XY and heading L1 loss for predicted trajectories, vehicle dynamics regression along with categorical cross-entropy for driving commands, supervision of the six NAVSIM proposal sub-scores, and mean cross-entropy over text tokens. Prompts and padding tokens are excluded from the language loss. Optimization uses AdamW (β1=0.9,β2=0.999\beta_1=0.9, \beta_2=0.999, weight decay = 0.01) with a cosine schedule decayed to 1e-6 across 64 NVIDIA A800 GPUs. Both the main text and Appendix will be unified with this schedule and loss formulation.


W2 — Does Bidirectional Attention Degrade Spatial Image Generation?

We think "temporal causality + intra-frame spatial bidirectionality" represents the standard paradigm in unified multimodal generation (e.g., Chameleon, Transfusion, VideoPoet), and is essential for preserving coherent 2D/3D spatial geometry. Enforcing an artificial causal mask across tokens within an individual image would instead disrupt geometric consistency and introduce directional artifacts.

We achieve both high generation quality and rigorous causal isolation through Block-wise Causal Masking:

  1. Block-wise Attention Mechanism: Unidirectional causal constraints are strictly applied across temporal steps and modal blocks (past frames cannot attend to future frames, and current planning is strictly prohibited from peeking at future latents or target answers). Meanwhile, within the same timestep, all tokens of an image or scene maintain full bidirectional attention to capture complete 2D/3D visual parallax and geometric context. High-fidelity visual/occupancy outputs are subsequently synthesized by the downstream SpatialDiT via global spatial denoising, completely free from causal scan constraints.

  2. Generative Quality Metrics: Under this design, the model achieves competitive performance across all generative tasks: image FID of 9.91, metric depth AbsRel of 0.13, and future 3D occupancy mIoU of 27.03 (geometric IoU of 61.49).

In summary, temporal causality does not necessitate autoregressive raster scanning within individual images. The detailed 2D attention mask diagram and visibility rules are formalized in Figure 6 and Section B of the Appendix in the revised manuscript.

Reviewer_4QHnOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes Doe-3, a World-Language-Action Model for autonomous driving that jointly predicts scene understanding outputs, future image frames, and planned trajectories to enable end-to-end autonomous driving. The method uses pretrained DriveTok and extends the model’s capabilities through different task heads. It achieves competitive results on mainstream benchmarks, indicating that future frame generation and text supervision can jointly improve planning.

Critical strengths

The unified framework for generation and understanding offers advantages in extensibility and downstream application as an autonomous driving solution. Extensive experiments demonstrate that text prediction and future frame generation jointly improve planning. The approach provides a unified solution and a useful experimental foundation for future work in the community.

Critical weaknesses

The role of language supervision has not been sufficiently validated. Although adding language improves planning scores, the relevant textual information is not used during inference. It therefore seems difficult to identify how the benefits of text supervision specifically manifest.

Questions for authors

Could the authors provide visual demonstrations together with a quantitative assessment of the accuracy of the language descriptions and an analysis of consistency among future predictions, predicted trajectories, and textual descriptions? Could the authors also show whether there are cases in which these three types of outputs contradict one another and explain how the model makes predictions in such cases? This evidence would help clarify the contribution of language supervision and assess the consistency of the model’s outputs.

4 comments
AuthorsOpenReview ↗

Author Response - Part 1

We sincerely thank the reviewer for the constructive feedback and encouraging evaluation. To address your concerns regarding the "mechanisms and quantitative accuracy of language supervision" and the "consistency analysis and conflict resolution across the tripartite outputs (future scene, planning trajectory, and text narration)," we have conducted extensive quantitative experiments and diagnostic analyses on a unified benchmark. Below are our point-by-point responses:


W1 & Q1 — Mechanism of Language Supervision

Core Mechanism: Language supervision restructures the multimodal representation space during training and functions as a directly reusable semantic context at inference.

  1. Semantic Alignment and Structuring in the Representation Space (Training Stage):
    Although the language generation head (LM Head) is bypassed during latency-critical planning deployment, textual supervision remains essential. As demonstrated in our ablation experiments (P achieves 87.5 PDMS, P+G achieves 90.0 PDMS, and incorporating language supervision via P+G+L advances to 94.0 PDMS), the language cross-entropy loss enforces a structured semantic mapping from low-level visual and geometric features to human-interpretable concepts (e.g., "traffic signal state," "lead-vehicle braking intent," "pedestrian crossing"). This implicit semantic alignment substantially improves the planning tokens' capacity to interpret complex interaction logic.
  2. Zero-Overhead Latent Space Reuse (Inference Stage):
    When historical context or upstream system instructions are present in an interactive driving scenario (e.g., high-level navigational commands or prior contextual descriptions), the planning tokens directly cross-attend to the hidden states of past text tokens under the causal self-attention mask, circumventing any autoregressive text decoding and re-encoding. Perturbation experiments demonstrate that perturbing historical prompt tokens induces a measurable dynamic response in the planning hidden states (mean absolute deviation of 6.6×10−26.6 \times 10^{-2}), verifying that semantic cues inform downstream planning representations directly.

W1 & Q2 — Quantitative Evaluation of Language Comprehension and Spatial Grounding Accuracy

To rigorously evaluate the language module's spatial grounding and scene reasoning capabilities, we conducted systematic benchmarks on 11,389 valid NAVTEST scenes across 44 recording sessions strictly disjoint from the training set. We evaluated two complementary capabilities: Spatial Visual Question Answering (VQA) and Scene Description Factuality.

All evaluated configurations shared an identical Qwen3-VL-4B backbone, received unified surround-view visual inputs, and operated without ground-truth target tokens; generated responses were parsed deterministically using rule-based evaluators with 100% test coverage. For spatial VQA, we report both standard Accuracy (Acc) and Balanced Accuracy (BAcc)—the unweighted mean of per-class recalls—to eliminate evaluation bias introduced by long-tail agent distributions and occluded lines of sight. For scene description factuality, we decomposed generated ego-behavior narrations and environmental descriptions into discrete atomic propositions (e.g., "a white vehicle is ahead," "the ego vehicle is turning right") and validated each against ground-truth 3D perceptual annotations, deriving factual Precision (P), Recall (R), and their harmonic mean (F1 score). The benchmark results are summarized in Table 1.

Table 1: Quantitative evaluation of spatial VQA and scene description factuality (NAVTEST, 11,389 scenes)

MethodNearest Agent Direction Acc / BAcc (%) ↑Nearest Agent Category Acc / BAcc (%) ↑VRU Direction Acc / BAcc (%) ↑Road Topology Layout Acc / BAcc (%) ↑Key Roadside Elements Acc / BAcc (%) ↑Description Factuality Fact P / R / F1 (%) ↑
Qwen3-VL-4B (Zero-shot)12.84 / 13.2379.77 / 78.6833.84 / 10.6266.93 / 78.2771.20 / 54.1083.23 / 44.33 / 57.85
Qwen3-VL-4B (Language SFT)87.44 / 87.0685.00 / 81.7483.87 / 62.9789.76 / 78.2783.60 / 52.7690.62 / 50.20 / 64.61
Doe-390.02 / 88.8494.30 / 93.2688.05 / 69.2690.55 / 87.1093.60 / 59.9591.50 / 54.27 / 68.13

Compared to the VLM baseline fine-tuned solely on driving text, Doe-3 outperforms across all six evaluation dimensions, demonstrating notable improvements in nearest-agent classification (Acc: 94.30% vs. 85.00%), roadside-element identification (Acc: 93.60% vs. 83.60%), and description factuality F1 (68.13% vs. 64.61%). These results confirm that joint geometric state modeling (DriveTok) and trajectory planning objectives effectively regularize the language module, anchoring its outputs in continuous physical space.

AuthorsOpenReview ↗

Author Response - Part 2

W1 & Q3: Consistency Analysis and Conflict Resolution Across Future Prediction, Planning Trajectory, and Textual Narration

1. Functional Specialization and Dynamic Controllability

The three output heads in Doe-3 perform distinct, decoupled roles:

  • Planning Head: Generates kinematically feasible trajectories from the shared latent space.
  • World Prediction Head: Synthesizes action-conditioned future scene rollouts (Figure 7, Appendix E.1).
  • Language Head: Decodes task-oriented VQA and high-level scene narrations.

To verify whether future generation complies strictly with commanded actions, we conducted a counterfactual trajectory intervention using four history frames to predict eight future frames (4 seconds). Visual Odometry (VO) displacement recovered from synthesized video frames was compared against commanded steering interventions and the original recorded log (Table 2).

Table 2: Displacement error of VO-recovered trajectories and logged ground truth relative to commanded counterfactual trajectories

Comparison Target1 s ↓ (m)2 s ↓ (m)3 s ↓ (m)1 to 3 s Mean ↓ (m)
VO-Recovered Trajectory vs. Commanded Action0.1790.6761.5830.813
Logged Ground Truth vs. Commanded Action0.7061.4292.1371.424

VO-recovered trajectories align closely with intervention commands, reducing mean displacement error by 42.9% relative to the recorded log. This demonstrates that future generation responds accurately to action inputs rather than memorizing historical sequences. In parallel, Doe-3 achieves 31.58 / 26.44 / 23.06 3D occupancy mIoU across 1 to 3 seconds (geometric IoU reaching 65.11 / 61.27 / 58.10), confirming both physical action responsiveness and spatial scene fidelity.


2. Tripartite Consistency Benchmark and Decision Arbitration

We evaluated cross-modal agreement on a diagnostic benchmark of 88 challenging interaction scenarios, including dense traffic, unmarked curves, and mixed pedestrian intersections. Across the planned trajectory (P), action-conditioned synthesized surround frames (S), and generated text narrations (T), we assessed turn intent (left, right, or straight) and motion state (accelerating, decelerating, or stationary). Synthesized scenes were categorized using standardized prompt evaluations. A scenario is marked consistent only when both attributes match completely (Table 3).

Table 3: Tripartite observable motion consistency benchmark under complex interaction scenarios (88 test cases)

ModelText vs. Trajectory (T vs. P)Text vs. Scene (T vs. S)Trajectory vs. Scene (P vs. S)Tripartite Mutual Consistency (T vs. P vs. S)
Doe-383/88 (94.3%)79/88 (89.8%)79/88 (89.8%)79/88 (89.8%)

Text and planning achieve 94.3% alignment, with overall tripartite agreement reaching 89.8%. For instance, in a curved driveway scenario (Figure 8, Appendix E.2), the planner executes a smooth right turn, the synthesized video depicts matching ego-motion and curb parallax, and the language head states "turn right and maintain a steady speed."

Discrepancy Arbitration and Safety Principles

When modal outputs disagree, Doe-3 enforces a strict safety hierarchy:

  • Exclusive Planning Authority: The trajectory produced by the planning head is executed as the sole control command.
  • Architectural Rationale: Text and video outputs are autoregressive generative processes vulnerable to sampling variance and hallucination. The trajectory planner is optimized under kinematic and safety boundaries (collision avoidance, drivable area compliance, headway time-to-collision). Language and generation remain supervisory and explanatory; they cannot override low-level actuation.

For example, in an intersection scenario flanked by parked cars (Figure 9, Appendix E.2), the planning head selects a right-turn path and the generated video depicts matching turning viewpoints, but autoregressive sampling in the language head erroneously outputs "continue straight and accelerate." Because control authority belongs entirely to the planning head, the system ignores the contradictory textual narration and safely executes the turn.

Thank you for the detailed response and additional experiments. The NAVTEST evaluation addresses my question about language factuality, and the cross-modal analysis clarifies how often the three outputs agree and which output governs execution when they disagree. The ablation supports the benefit of language supervision for planning. I maintain my weak accept recommendation.

Thank you for your kindly feedback and for taking the time to read our response. We’re glad our additional experiments and clarifications addressed your concerns.

Reviewer_psV1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Doe-3, a unified World-Language-Action (WLA) framework for autonomous driving that integrates scene understanding, future prediction, and trajectory planning. The method builds on pretrained DriveTok representations and uses a shared autoregressive transformer to model interactions among visual scene tokens, language, and actions. A lightweight diffusion head is introduced for action-conditioned future scene generation, while task-specific heads support language understanding and trajectory planning. The framework is trained through a three-stage procedure with multiple learning objectives. Experiments on NAVSIM v1/v2 and nuScenes show strong performance in trajectory planning, depth prediction, and occupancy forecasting. Ablation studies further suggest that incorporating future generation and language supervision improves planning performance.

Critical strengths
  1. Strong experimental performance. The method achieves competitive results on both NAVSIM benchmarks, with particularly large improvements in ego progress.

  2. A useful unified framework. Integrating language understanding, trajectory planning, and future-scene generation within a shared spatial representation is a meaningful direction. The multi-task ablations provide initial evidence that generation and language supervision benefit planning.

Critical weaknesses
  1. Limited methodological novelty and unclear cross-task interaction. The framework mainly combines pretrained DriveTok representations, a Qwen3-VL backbone, and task-specific planning and diffusion heads. The main novelty beyond existing unified driving models is not sufficiently established. More importantly, the planning tokens do not appear to use future predictions generated under candidate actions during the current planning step. It is therefore unclear whether the model learns a genuinely coupled world-language-action mechanism or mainly benefits from auxiliary supervision.

  2. Insufficient evidence for the central claims. While the planning results are strong, the evaluation does not fully establish the role of language understanding and world prediction. There is no quantitative evaluation of language grounding or reasoning, and counterfactual future generation is mainly demonstrated qualitatively. The ablations also do not isolate the effects of the shared representation, architectural choices, and additional training objectives.

Questions for authors
  1. What is the main methodological difference between Doe-3 and existing unified models such as UniDrive-WM, DriveLaW, and other world-language-action frameworks?

  2. Since planning tokens precede the current action-conditioned world tokens, how does the planner explicitly benefit from predicted future states at inference time?

  3. Does the planning module receive generated scene descriptions at inference time, or is language supervision used only during training? How is this implemented?

  4. Could the authors provide quantitative results for language understanding, VQA, and spatial grounding? Figure 5 alone is insufficient to evaluate these capabilities.

  5. How much of the planning improvement comes from the pretrained DriveTok representation rather than joint training? A controlled comparison using the same backbone and representation would be helpful.

  6. How are the language descriptions and VQA annotations obtained? Are they human-annotated or generated by another model, and how is their quality verified?

  7. Can the authors quantitatively evaluate counterfactual consistency under different ego actions? How well does the generated future reflect changes in vehicle motion and surrounding traffic?

  8. The improvement on NAVSIM v1 appears to be driven largely by ego progress, while some safety-related metrics remain below those of competing methods. Could the authors analyze this trade-off, especially in challenging scenarios?

  9. Section 3.3 describes joint optimization of language, planning, and generation, whereas Appendix A states that the final stage uses only planning and future-generation objectives. Which objectives are active in each stage? The detailed loss weights and training targets should also be reported.

  10. What are the inference latency, GPU memory requirements, and computational costs of the complete framework? Can the planning module run efficiently without executing the diffusion head?

8 comments
AuthorsOpenReview ↗

Author Response - Part 1

We sincerely thank the reviewer for the thorough reading and constructive feedback. Below is our response detailing the architectural distinctions of Doe-3.

W1 & Q1: Structural differences between Doe-3 and unified driving models

The core contribution of Doe-3 is to build a unified driving model with a driving-oriented scene representation. Driving-native end-to-end models have proven the effectiveness and efficiency of using 3D scene representations (e.g., bird's eye view) [UniAD, DiffusionDrive]. However, most existing vision-language-action and world-action models use view-wise visual encoders or variational autoencoders (VAE) pretrained with general image data. This also results in a discrepancy of driving modeling even for most unified driving models, e.g., using an understanding encoder for perception tasks and a generation encoder for world modeling [FSDrive]. We bridge this gap by using a shared world represetnation to capture the apperance, geometry, and semantics of the surrounding 3D scenes. Driving can then be formulated by a unified autoregresive process of world tokens (scene modeling), language tokens (human interaction), and action tokens (planning), which is instantiated by our Doe-3 model.

Our contribution lies in this formulation, which connects scene understanding, world modeling, human interaction, and planning through shared world tokens and a unified autoregressive process. Table 1 systematically compares Doe-3 with representative unified driving models across four design dimensions.

Table 1: Systematic comparison between Doe-3 and representative unified driving models

MethodScene RepresentationRole of LanguageFuture Prediction and Planning InterfacePredicted Future Modalities
UniDrive-WMVLM visual tokensScene understanding and VQAPlanned trajectory conditions future generation; joint prediction supervises understandingRGB
DriveLaWVideo generator latentsNot included in the planner interfaceVideo features feed directly into the diffusion plannerRGB video
FSDriveVisual tokens and imagined framesInstruction following and visual chain-of-thoughtImagined future frames serve as intermediate reasoning steps for inverse dynamics planningRGB frames with sparse 2D/3D priors
DriveVLA-W0Discrete or continuous image featuresContextual instruction inputDense training supervision via world modeling; lightweight action experts for deploymentRGB
Doe-3Shared 3D DriveTok geometric latentsSequential tokens causally accessible to planning queriesPredicts action-conditioned future states; planning precedes generation, enabling generation/language branch skipping at inferenceUnified latent decoding into RGB, metric depth, semantics, and 3D occupancy

Three fundamental structural distinctions set Doe-3 apart:

  1. Unified 3D geometric state space for observations and predictions. While existing works use 2D image tokens (UniDrive-WM, DriveVLA-W0) or general video latents (DriveLaW), Doe-3 organizes all tasks around a shared 3D geometric latent space (DriveTok). The same representation serves simultaneously as current observation features and future prediction targets, eliminating the representational gap between perception and world modeling.
  2. Direct causal semantic flow with decoupled inference. Unlike models that require generating verbal rationales during control (FSDrive), Doe-3 allows planning queries to attend directly to historical language hidden states through causal masking. During real-time deployment, the generation and language branches can be completely bypassed, decoding planning trajectories directly to achieve 1.77$\times$ speedup and low latency without sacrificing control accuracy.
  3. Dense multi-modal future decoding from a single latent. While prior world models predict only RGB pixels or sparse bounding boxes, a single predicted DriveTok latent in Doe-3 jointly decodes into multi-view RGB, metric depth, semantic segmentation, and full 3D occupancy grids, providing physically grounded spatial constraints for safe planning.

In short, Doe-3 provides a unified 3D geometric state, direct causal text-to-plan interaction, and dense physical world generation. Detailed operational sequences, latency benchmarks, and comprehensive empirical comparisons are reported in Table 2 through Table 9.

AuthorsOpenReview ↗

Author Response - Part 2

W1 & Q2 — Why does planning not need to wait for future generation?

Our design prioritizes inference efficiency. Fast-WAM from robotics shows that world modeling benefits policy learning during training as a future imagination task, while explicit future generation can be bypassed at inference. Following this principle, Doe-3 jointly learns world modeling and planning during training but omits future generation at deployment, predicting actions directly from the shared representations. The inductive bias gained from world model training remains preserved within the shared weights, and driving decisions are returned immediately without waiting for future latent synthesis.

To validate this choice, we benchmarked Doe-3 against a "predict-future-first, then plan" baseline where the planner explicitly attends to predicted future tokens. Both architectures take 4-frame history inputs and run on a single NVIDIA A800-80GB GPU at batch size 1. The future-first baseline conditions on the latest observed action, generates a future latent via 10-step denoising, and then executes planning. Table 2 reports the latency comparison.

Table 2: Latency comparison between direct planning and "predict-future-first" planning (A800-80GB, batch size 1)

ArchitectureLatency (ms) ↓FPS ↑
Doe-3: Direct planning816.011.225
Predict future tokens first, then plan1443.640.693

Direct planning reduces runtime latency from 1443.64 ms to 816.01 ms (a 43.5% reduction, or a 1.77× speedup), while achieving 94.0 PDMS / 90.8 EPDMS in benchmark evaluations. These results validate our architectural decision: leveraging future prediction as a representational prior during training enables direct, low-latency decision-making at test time without compromising planning performance.


W1 & Q3 — How does language enter the planner at inference time?

Language operates through two complementary mechanisms in our architecture:

  • Training stage: Language cross-entropy loss updates the shared backbone and scene interfaces, aligning high-level semantic representations with spatial geometry and control.
  • Inference stage: In the causal sequence, historical scene descriptions and contextual text precede planning tokens (ptp_t). The planning tokens directly cross-attend to the hidden states of these text tokens. These representations are reused directly in the latent space without requiring intermediate string decoding and re-tokenization.

Generating new language is treated as a separate, on-demand operation: when a description or VQA response is explicitly queried, the LM head autoregressively decodes new tokens. Planning operates on available historical context without needing to generate new descriptive sentences prior to every control step. Supervised target answers and future states are causally masked and never leak into planning inputs.

AuthorsOpenReview ↗

Author Response - Part 3

W2 & Q4 — Quantitative results on language understanding and spatial grounding

We conducted two quantitative evaluations on model-generated text: spatial grounding question answering (VQA) and factual accuracy of scene descriptions. All models share the Qwen3-VL-4B backbone and take identical 8-camera surround-view inputs with deterministic cropping.

  • Benchmark dataset & annotations: The VQA evaluation comprises 11,389 valid scenes from NAVTEST across 44 distinct driving sessions strictly disjoint from the training split. Ground-truth answers are programmatically generated from 3D annotations at the current frame and verified for camera visibility; if multiple objects share identical distances or directions, all are preserved as valid answers. Vulnerable Road Users (VRUs) encompass pedestrians and cyclists. Scene descriptions are decomposed into atomic, verifiable factual propositions (e.g., object presence, relative directions, road layout, roadside elements) and verified against ground-truth facts extracted from 3D scene graphs and verified by blind human visual inspection.
  • Evaluation metrics: VQA reports Accuracy (Acc, percentage of answers matching any valid label) and Balanced Accuracy (BAcc, mean recall across categories computed on single-label questions). Scene descriptions report factual Precision (P), Recall (R), and F1 score (harmonic mean of P and R). All metrics are percentages (higher is better).

Table 3: Spatial grounding VQA and scene description factual accuracy on NAVTEST (11,389 scenes)

MethodNearest Agent Direction Acc / BAcc (%) ↑Nearest Agent Category Acc / BAcc (%) ↑VRU Direction Acc / BAcc (%) ↑Road Layout Acc / BAcc (%) ↑Roadside Elements Acc / BAcc (%) ↑Scene Description Fact P / R / F1 (%) ↑
Qwen3-VL-4B (Zero-shot)12.84 / 13.2379.77 / 78.6833.84 / 10.6266.93 / 78.2771.20 / 54.1083.23 / 44.33 / 57.85
Qwen3-VL-4B (Language SFT)87.44 / 87.0685.00 / 81.7483.87 / 62.9789.76 / 78.2783.60 / 52.7690.62 / 50.20 / 64.61
Doe-390.02 / 88.8494.30 / 93.2688.05 / 69.2690.55 / 87.1093.60 / 59.9591.50 / 54.27 / 68.13

Table 3 highlights three key findings:

  1. Doe-3 consistently outperforms both baselines across all six metrics. Compared to the supervised fine-tuned (SFT) baseline, Doe-3 improves nearest agent category accuracy from 85.00% to 94.30%, roadside element accuracy from 83.60% to 93.60%, road layout BAcc from 78.27% to 87.10%, and description F1 from 64.61% to 68.13%.
  2. Performance gains stem from shared 3D geometry and joint training. Both the SFT baseline and Doe-3 utilize the same backbone and text supervision; the crucial difference is that Doe-3 jointly optimizes world modeling and planning losses over DriveTok 3D latents. The substantial improvements in directional and categorical tasks confirm that 3D geometric tokens provide critical spatial anchoring for language grounding.
  3. Description gains are primarily driven by recall. While all models maintain high precision, Doe-3 increases factual recall from 50.20% to 54.27%, capturing more scene context without hallucinating false propositions.

W2 & Q5 — Do planning gains originate from representation or joint modeling?

Table 4 combines the training objective ablations with representation replacement experiments. The P, P+G, and P+G+L variants share the same DriveTok representation and scene token interface with matched parameter budgets. The VGGT-based model preserves the backbone and planning head of the P configuration, swapping only the input scene representation.

Table 4: Ablation on training objectives and scene representations (NAVSIM)

ModelRepresentationNAVSIM v1 PDMS ↑NAVSIM v2 EPDMS ↑
PDriveTok87.586.2
P+GDriveTok90.088.8
P+G+LDriveTok94.090.8
VGGT-BasedVGGT78.1677.79

Table 4 reveals that both representation and joint modeling contribute critically at distinct levels:

  • Representation establishes the baseline foundation: Under identical backbones, planning heads, and objectives (P only), replacing DriveTok with VGGT degrades PDMS from 87.5 to 78.16 (-9.34) and EPDMS from 86.2 to 77.79 (-8.41). This confirms that geometry-aware 3D latents provide an essential inductive foundation for planning.
  • Joint modeling yields progressive, cumulative gains: Adding the generative world prediction objective (P → P+G) yields +2.5 PDMS / +2.6 EPDMS; further incorporating language supervision (P+G → P+G+L) adds +4.0 PDMS / +2.0 EPDMS. Combined multi-task training provides an overall boost of +6.5 PDMS / +4.6 EPDMS over planning alone.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.