- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Doe-3 is a unified world-language-action model. Its core method encodes synchronized multi-view observations into scene tokens that preserve appearance, semantics and geometry via a pre-trained driving world representation. It unifies these tokens with language and action modalities through a shared autoregressive backbone, and equips a lightweight diffusion head to map world tokens to multi-modal future scene outputs. The model is progressively aligned and optimized in three stages, realizing deep coupling between semantic understanding and scene dynamics.
Critical strengths
The framework proposed by the authors is relatively concise, and it effectively unifies multiple modalities. Its core advantages are as follows:
- Unified multi-modal representation eliminates redundant independent visual encoders, integrating multi-view scene tokens, language and driving actions into one shared autoregressive backbone, breaking long-existing representation barriers across perception, prediction and planning.
- Natively coupled multi-task learning connects semantic understanding directly with scene dynamics via joint language, planning and future prediction objectives, supporting grounded language response, safe trajectory planning and action-conditioned future forecasting in a single forward pass.
Critical weaknesses
- The proposed method is relatively concise, but there are many difficulties in training such a model. The authors proposed a three-stage training framework, but I haven't seen more details about these three stages of training, such as what kind of data is used in each stage. As the objectives of each stage are different, I think that the data and the training strategies used should also differ.
- From the proposed framework, all image tokens, text tokens, and other tokens are intertwined into a unified sequence, where each token can only access itself and all tokens preceding it. However, for the tokens of a single image, they can see each other. Will this paradigm have a negative impact on image generation?
Questions for authors
Please see weaknesses above.
Author Response
We sincerely thank the reviewer for the encouraging remarks and constructive feedback. Below, we provide the clarified details of the three-stage training schedule and explain how our attention design simultaneously ensures temporal causality and full spatial interaction.
W1 — Three-Stage Training: Data, Modules, and Objectives
Thank you for pointing out the inconsistency. Appendix A inadvertently omitted the language objective in the final stage, which will be corrected. The clarified training schedule is summarized in Table 1.
Table 1: Three-stage progressive optimization schedule. P, G, and L denote planning, future world generation, and language objectives, respectively.
In Stage 1, the backbone and downstream task heads are frozen, optimizing only the scene interface via planning and generative world-modeling losses. Stage 2 adapts the shared backbone via LoRA while introducing language supervision. Stage 3 jointly fine-tunes the interfaces, shared LoRA adapters, and designated task modules while retaining all three training objectives. All stages utilize both OpenScene and nuScenes for the epochs specified in Table 1, computing supervisory losses according to available task annotations.
The composite loss function is defined as:
L=3Llatent+4Lflow+0.5Ltraj+Ldyn+0.75Lscore+λLLCE
The terms represent, respectively: future latent MSE, flow matching on latent velocity fields, XY and heading L1 loss for predicted trajectories, vehicle dynamics regression along with categorical cross-entropy for driving commands, supervision of the six NAVSIM proposal sub-scores, and mean cross-entropy over text tokens. Prompts and padding tokens are excluded from the language loss. Optimization uses AdamW (β1=0.9,β2=0.999, weight decay = 0.01) with a cosine schedule decayed to 1e-6 across 64 NVIDIA A800 GPUs. Both the main text and Appendix will be unified with this schedule and loss formulation.
W2 — Does Bidirectional Attention Degrade Spatial Image Generation?
We think "temporal causality + intra-frame spatial bidirectionality" represents the standard paradigm in unified multimodal generation (e.g., Chameleon, Transfusion, VideoPoet), and is essential for preserving coherent 2D/3D spatial geometry. Enforcing an artificial causal mask across tokens within an individual image would instead disrupt geometric consistency and introduce directional artifacts.
We achieve both high generation quality and rigorous causal isolation through Block-wise Causal Masking:
Block-wise Attention Mechanism: Unidirectional causal constraints are strictly applied across temporal steps and modal blocks (past frames cannot attend to future frames, and current planning is strictly prohibited from peeking at future latents or target answers). Meanwhile, within the same timestep, all tokens of an image or scene maintain full bidirectional attention to capture complete 2D/3D visual parallax and geometric context. High-fidelity visual/occupancy outputs are subsequently synthesized by the downstream SpatialDiT via global spatial denoising, completely free from causal scan constraints.
Generative Quality Metrics: Under this design, the model achieves competitive performance across all generative tasks: image FID of 9.91, metric depth AbsRel of 0.13, and future 3D occupancy mIoU of 27.03 (geometric IoU of 61.49).
In summary, temporal causality does not necessitate autoregressive raster scanning within individual images. The detailed 2D attention mask diagram and visibility rules are formalized in Figure 6 and Section B of the Appendix in the revised manuscript.