acceptodds
Under review as a conference paper at ICLR 2027

DriveUniTok: A Unified Multimodal Tokenizer for Driving World-Action Models

Abstract

World models are emerging as a fundamental component of policy learning for real-world agents, enabling agents to predict future world states and plan actions within a learned representation of the dynamic environment. Existing approaches, however, often model appearance, metric geometry represented by projected LiDAR depth, and ego motion either partially or independently, overlooking the fact that these modalities are different projections of a shared underlying world state. This separation can lead to information loss and cross-modal inconsistency, ultimately limiting downstream performance. Motivated by this unified view of the world, we propose DriveUniTok, a unified multimodal tokenizer for driving world-action models (WAMs) that encodes these modalities in a single decodable continuous latent. We freeze DriveUniTok and adapt a pretrained diffusion model, DriveUniWAM, to predict future unified latents; the tokenizer then decodes them into future appearance, projected LiDAR depth, and ego trajectories. Experiments on NAVSIM demonstrate that our unified design achieves better future prediction and planning performance at lower downstream computational cost than separate-stream alternatives. With reinforcement learning post-training, DriveUniWAM achieves a PDMS of 91.54, reaching state-of-the-art performance among the evaluated WAM-based planners. The anonymous code is available at https://anonymous.4open.science/r/DriveUniTok-B358/.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Autonomous Driving, World Models, World-Action Models, Multimodal Tokenization, Unified Latent Representations, Video Diffusion Models

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_AUSHOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

DriveUniTok encodes front-camera RGB, projected LiDAR depth, and clip-relative ego pose into one continuous latent of 584 tokens with width 16 per five-frame clip. Shared queries aggregate the modalities, while modality-specific readouts reconstruct them. The frozen tokenizer is coupled to a pretrained video diffusion backbone for future-latent prediction and then refined with Flow-GRPO for planning. The authors report 91.0 supervised PDMS, 91.54 after reinforcement learning, and 90.3 EPDMS on NAVSIM-v2.

Critical strengths

S1. The shared bottleneck addresses a relevant representation question: whether visual, geometric, and ego-motion signals can be jointly encoded before driving-world prediction.

S2. Table 2 provides natural- and equal-budget controls. At equal downstream cost, PDMS rises from 87.5 to 91.0, with lower future prediction errors. Relative to the natural-budget control, per-step DiT computation falls from 34.2 to 16.3 TFLOPs. Results across five backbones and on NAVSIM-v2 add useful breadth.

Critical weaknesses

W1. The comparison changes more than latent organization. Parameter sharing, cross-modal connectivity, decoder visibility, and ego readout also differ (Section 4.3; Appendix A). Table 2 therefore supports the complete design, but does not isolate the benefit of joint tokenization.

W2. The claimed cross-modal consistency is not measured directly. LPIPS/FVD, depth error, and trajectory error assess outputs separately; they do not show whether modalities agree within the same predicted future. The language in Sections 3.2–3.3 should be qualified or supported by a direct consistency measure.

W3. Test-time planning settings are insufficiently specified. Appendix C.2 describes sampling during RL training, but the denoising steps, candidate count, trajectory selection rule, and any additional scorer used for evaluation remain unclear. This limits reproducibility and makes the computational comparison hard to interpret.

W4. The smaller planning gains lack uncertainty estimates. The 0.54 PDMS increase after RL coincides with lower NC (99.0 to 98.7) and TTC (96.3 to 95.6) in Table 3. Seed variation or paired intervals would clarify whether these changes are reliable and how to interpret the safety-metric trade-off.

Questions for authors

Q1. Can the authors compare against separate encoders with unrestricted shared decoding, and report a direct geometric or motion-consistency measure for generated futures?

Q2. What are the test-time denoising budget, candidate count, and trajectory selection procedure for each main table? Do the supervised and RL models use identical settings?

Q3. How much do the principal planning results vary across seeds or paired evaluation runs? Are the NC and TTC decreases after RL consistent?

4 comments
AuthorsOpenReview ↗

Response to Reviewer (1/2): Architectural controls (W1, Q1) and cross-modal consistency (W2, Q1)

We thank the reviewer for the constructive and precise suggestions. Both requested experiments have been added to the revised manuscript.

W1 / Q1a. Separate encoders with unrestricted shared decoding, and further controls

Table 2 compares the complete designs. To attribute the gain to specific factors (latent organization, parameter sharing, cross-modal connectivity, decoder visibility, and ego readout), we add the exact control the reviewer requested (B), together with two joint-encoder controls. All variants use the same 584-token budget, latent width 16, frozen DINOv3-L features, objectives, DriveUniWAM backbone, and inference settings:

VariantWhat changes from the previous rowgFVD ↓AbsRel ↓ADE ↓PDMS ↑
A. Separate (Equal Budget)–12.800.18500.418687.5
B. Separate Enc. + Shared Dec.decoder reads all latent tokens, shared ego readout9.450.14200.395088.8
C-mask. Joint Enc. (Masked)one shared encoder; partitioned queries (288 RGB / 288 depth / 8 ego), no cross-group attention8.920.13850.402089.2
C. Joint Enc. (Full Attn.)only the attention mask is removed7.350.11800.365090.1
D. Unified (Ours)576 shared visual queries on a common spatial grid6.610.10590.381691.0
  • Answer to Q1a. Unrestricted shared decoding (B) recovers part of the gap (87.5 → 88.8 PDMS). However, the full unified design is 2.2 PDMS points higher (91.0) and roughly 30% better in gFVD (9.45 → 6.61). Decoder-level sharing therefore does not explain the benefit of joint tokenization.
  • Cross-modal interaction is isolated. C-mask → C changes only the attention mask, with identical parameters and query allocation. It improves PDMS by 0.9, which exceeds the seed standard deviation of 0.15 (Appendix G.4).
  • Shared queries add further gains. At the same parameter count, C → D further improves gFVD, depth AbsRel, and PDMS. This step introduces shared visual queries and a unified spatial reference together, so it evaluates their combined effect.
  • Capacity. The joint-encoder variants have 109M fewer tokenizer parameters than A (847.3M vs. 956.6M), so the gains are not due to capacity.

W2 / Q1b. Direct cross-modal consistency of generated futures

Following the reviewer's suggestion, we measure whether the modalities agree within the same predicted future, using each model's own generated RGB, depth, and ego motion:

  • Depth–Ego Geometric Consistency (↓) follows SC-SfMLearner [1]: it is the normalized discrepancy between the generated depth of frame t warped by the predicted ego motion and the generated depth of frame t+1.
  • RGB–Depth–Ego Rigid Flow EPE (↓) is inspired by GeoNet [2]: it is the endpoint error between RAFT [3] flow on the generated RGB and the rigid flow implied by the generated depth and ego motion.

All methods are scored on the same set of valid static pixels on NAVSIM navtest, defined by a GT-based static and visibility mask intersected with every method's validity mask.

MethodGeometric Consistency ↓Rigid Flow EPE ↓
GT (reference)0.023442.370 px
Separate (Equal Budget)0.049219.745 px
Separate (Natural Budget)0.047469.622 px
Unified (Ours)0.025845.622 px

Unified reduces the two errors by 47.5% / 42.3% relative to equal budget, and by 45.6% / 41.6% relative to the natural budget, which uses about twice the downstream tokens. Its geometric consistency is close to the GT reference.

Qualified claims. As the reviewer suggested, we have revised the language. The abstract and Section 3.2 now describe the benefit specifically as static-region geometric consistency and point to the direct measurement.

In the revised manuscript, the controls are reported in Table 3 and Appendix G.1, and the consistency results and protocol in Table 4 and Appendix G.2.

[1] Bian et al., SC-SfMLearner, NeurIPS 2019. [2] Yin & Shi, GeoNet, CVPR 2018. [3] Teed & Deng, RAFT, ECCV 2020.

AuthorsOpenReview ↗

Response to Reviewer (2/2): Test-time protocol (W3, Q2) and seed robustness (W4, Q3)

W3 / Q2. Test-time denoising budget, candidates, and trajectory selection

The test-time protocol is identical for every model in the main prediction and planning tables (Tables 2–6), including all architectural variants:

SettingValue
Sampler10 deterministic Euler steps, one DiT forward pass per step
Candidates per scene1 (a single joint sample containing two future clips)
Trajectory selectionNone: no candidate reranking, no additional inference-time scorer
OutputDecoded clip-relative poses composed into 8 waypoints at 0.5 s over 4 s, submitted directly to NAVSIM
SFT vs. RLIdentical settings

The G = 8 stochastic Flow-SDE samples in Appendix C.2 are used only during RL training to compute group-relative advantages. They are never used for test-time selection, and PDMS is computed only for evaluation. The SFT → RL gain is therefore measured at the same single-sample inference budget.

This also makes the computational comparison straightforward to interpret. One planning decision costs 172.12 TFLOPs, of which 163.29 TFLOPs is the 10-step DiT. Measured end to end, it takes 1,908 ms on an H20 and 1,363 ms on an RTX 4090. The SFT and RL models have the same cost per decision.

In the revised manuscript, these settings are stated in Section 4.1 and Appendix C.3, the RL training details are in Appendix C.2, and the full cost and latency breakdown is in Appendix G.5.

W4 / Q3. Seed variation and paired evaluation

We trained three independent SFT runs (seeds 0, 1, 42) and initialized each RL run from its matched SFT checkpoint. The frozen tokenizer and evaluation settings were shared, and every checkpoint was evaluated on the full NAVSIM navtest set.

MetricSFT (mean ± std)SFT + RL (mean ± std)Paired Δ (RL − SFT)Per-seed Δ
PDMS ↑90.97 ± 0.1591.55 ± 0.05+0.58 ± 0.20+0.80 / +0.40 / +0.54
EP ↑85.57 ± 0.3587.37 ± 0.25+1.80 ± 0.10–
NC ↑99.00 ± 0.1098.70 ± 0.10−0.30 ± 0.20−0.50 / −0.10 / −0.30
TTC ↑96.30 ± 0.2095.60 ± 0.20−0.70 ± 0.00−0.70 in each run

Entries are mean ± sample standard deviation over three seeds; paired differences are computed within each seed.

  • The PDMS gain is reliable. RL improves PDMS in every seed (+0.80 / +0.40 / +0.54). The mean gain of +0.58 is nearly 4× the seed-to-seed standard deviation of the SFT model (0.15), and the RL models are even more stable across seeds (std 0.05). The improvement therefore clearly exceeds training variance.
  • Interpreting the NC/TTC change: RL makes the policy less conservative. The NC and TTC decreases occur in every seed, so they reflect a systematic change in driving behavior rather than noise. The supervised model is conservative: its ego progress (85.6) is below the human reference (87.5). Our RL reward follows the PDM score form with the NC and DAC gates intact, and, following ReCogDrive, gives ego progress twice its evaluation weight (EP : TTC : Comfort = 10 : 5 : 2; Appendix C.2). RL post-training therefore shifts the policy toward more assertive driving. Ego progress rises to 87.4, essentially matching the human level, at the cost of slightly smaller safety margins. Since TTC flags any predicted time-to-collision below a fixed threshold, closer following and earlier merging are the first behaviors to register there. This is the expected outcome of optimizing a composite planning reward, and the net effect is positive, with PDMS improving in all three seeds. In absolute terms the safety metrics stay high: NC remains at 98.7, comparable to the strongest planners in Table 5.

In the revised manuscript, these results are in Appendix G.4 (Table 12), and the matched RL control for the separate representation is in Table 11.

AuthorsOpenReview ↗

Summary of our responses to Reviewer AUSH

Dear Reviewer AUSH,

Thank you again for your precise and constructive questions. Below is a brief map from each point to our response and the revised manuscript (changes in blue):

  • W1 / Q1 (separate encoders with unrestricted shared decoding): Added exactly this control (B), plus two joint-encoder controls. B reaches 88.8 PDMS vs. 91.0 for Unified, so decoder-level sharing does not explain the gain. → Response 1/2; Table 3, Appendix G.1.
  • W2 / Q1 (direct consistency measure): Depth–ego reprojection consistency and RGB–depth–ego rigid-flow EPE on generated futures, with errors reduced by 47.5% / 42.3%. The claims in the abstract and Section 3.2 are now qualified to static-region geometric consistency. → Response 1/2; Table 4, Appendix G.2.
  • W3 / Q2 (test-time protocol): 10 Euler steps, a single sample, no reranking or extra scorer, identical for SFT and RL. G = 8 samples are used only during RL training. → Response 2/2; Section 4.1, Appendix C.3.
  • W4 / Q3 (seed variation, NC/TTC): Three paired seeds give a PDMS gain of +0.58 ± 0.20, positive in every seed. The consistent NC/TTC decrease reflects a less conservative policy, with ego progress rising to human level under a progress-weighted reward (Appendix C.2). → Response 2/2; Appendix G.4.

We would be grateful to know whether these address your concerns, or if any point needs further clarification.

AuthorsOpenReview ↗

Follow-up on our responses

Dear Reviewer AUSH,

Thank you again for your precise and constructive questions. We have added every experiment you requested: the separate-encoder / shared-decoder control, direct consistency measures for generated futures, the full test-time protocol, and three-seed paired results with an analysis of the NC/TTC change (Responses 1/2–2/2, summarized above).

As the discussion period is drawing to a close, we would greatly appreciate your feedback. If our responses have resolved your concerns, we would be sincerely grateful if you could reconsider your score in light of the new results. If any point remains unclear, we would be glad to clarify further.

Best regards,

The Authors

Reviewer_WSCSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes DriveUniTok, a unified multimodal tokenizer that jointly encodes RGB, projected LiDAR depth, and ego poses into a shared continuous latent space. Based on this representation, DriveUniWAM employs flow matching for joint future-scene prediction and trajectory planning, followed by GRPO-based reinforcement learning. Experiments on NAVSIM demonstrate competitive planning performance, achieving 91.54 PDMS.

Critical strengths
  1. The paper presents a unified framework for multimodal representation learning and world-action modeling.
  2. The proposed method achieves competitive results on NAVSIM with improved downstream computational efficiency.
Critical weaknesses
  1. Insufficient validation of the core motivation. The paper claims that unified tokenization improves cross-modal consistency, but the experiments only report individual prediction and planning metrics without directly evaluating such consistency.

  2. Inadequate controlled comparisons. The unified and separate baselines in Table 2 differ in query allocation, parameter sharing, and cross-modal connectivity, making it difficult to isolate the actual benefits of unified tokenization.

  3. Limited novelty. The method mainly integrates existing techniques, including shared-query attention, temporal compression, flow matching, and GRPO, without substantial methodological innovation.

  4. Incomplete efficiency evaluation. The reported computational savings focus on the DiT backbone, while the end-to-end inference cost, including multimodal encoding and decoding, is not evaluated.

Questions for authors

I would like to see stronger experimental evidence and more rigorous analysis addressing the concerns raised above.

9 comments
AuthorsOpenReview ↗

Response to Reviewer (1/4): Direct evaluation of cross-modal consistency (W1)

We thank the reviewer for suggesting a direct evaluation of cross-modal consistency. We introduce two geometry-based metrics that measure whether the RGB, depth, and ego motion of the same generated future agree with each other:

  • Depth–Ego Geometric Consistency (↓): Following SC-SfMLearner [1], each valid source pixel is back-projected with the generated depth, transformed by the predicted ego motion, and projected into the next frame; we compute the normalized discrepancy between the transformed depth and the generated depth at the corresponding pixel.
  • RGB–Depth–Ego Rigid Flow EPE (↓): Inspired by GeoNet [2], we compute the endpoint error between optical flow estimated from the generated RGB frames by pretrained RAFT [3] and the rigid flow implied by the generated depth and ego motion.

Protocol. Each model is evaluated on NAVSIM navtest using its own generated RGB, depth, and ego motion. Since rigid flow is valid only for static content, all methods are scored on the same set of static, valid pixels, defined by a GT-based mask. The GT row applies the same protocol to real sensor data and indicates the residual error of the measurement itself. Full details are given in Appendix G.2.

MethodGeometric Consistency ↓Rigid Flow EPE ↓
GT (reference)0.023442.370 px
Separate (Equal Budget)0.049219.745 px
Separate (Natural Budget)0.047469.622 px
Unified (Ours)0.025845.622 px

Compared with the equal-budget separate baseline, DriveUniTok reduces geometric inconsistency by 47.5% and rigid-flow EPE by 42.3%. Compared with the natural-budget baseline, which uses about twice the downstream tokens, the reductions are 45.6% and 41.6%. The geometric consistency of DriveUniTok (0.02584) is close to the GT reference (0.02344). These results provide direct evidence that the jointly generated modalities are more mutually consistent in static regions. In the revised manuscript, these results are reported in Table 4 (Section 4.3), and the formulations and evaluation protocol are given in Appendix G.2.

[1] Bian et al., SC-SfMLearner, NeurIPS 2019. [2] Yin & Shi, GeoNet, CVPR 2018. [3] Teed & Deng, RAFT, ECCV 2020.

AuthorsOpenReview ↗

Response to Reviewer (2/4): Controlled architectural comparisons (W2)

We thank the reviewer for this suggestion. Table 2 compares the complete unified and separate designs. To attribute the gain to specific design choices (latent organization, parameter sharing, and cross-modal connectivity), we add three intermediate variants that introduce these factors one at a time. All variants share the same 584-token budget, latent width (16), downstream sequence length (1,752), frozen DINOv3-L features, training data, objectives, DriveUniWAM backbone, and inference settings:

  • A. Separate (Equal Budget): the original equal-budget control, with separate encoders and decoders (288 RGB / 288 depth / 8 ego queries).
  • B. Separate Encoders + Shared Decoder: keeps the separate encoders of A, but the shared decoder reads all latent tokens and uses a shared ego readout. A → B tests shared decoding.
  • C-mask. Joint Encoder (Masked Attention): a single shared encoder with modality-partitioned queries (288 RGB / 288 depth / 8 ego). Each visual group reads only its own modality, and an attention mask blocks interaction between groups. B → C-mask tests joint encoding.
  • C. Joint Encoder (Full Attention): identical to C-mask, including parameters and query allocation, except that the mask is removed. C-mask → C tests cross-modal query interaction, and it is the only change between the two.
  • D. Unified (Ours): 576 shared visual queries that aggregate both RGB and depth through sequential deformable cross-attention on a common spatial grid, plus 8 ego queries. C → D tests the shared-query design: shared visual queries together with a unified spatial reference.
VariantParams (M)gFVD ↓Depth AbsRel ↓Traj. ADE ↓PDMS ↑
A. Separate (Equal Budget)956.612.800.18500.418687.5
B. Separate Enc. + Shared Dec.907.89.450.14200.395088.8
C-mask. Joint Enc. (Masked)847.38.920.13850.402089.2
C. Joint Enc. (Full Attention)847.37.350.11800.365090.1
D. Unified (Ours)847.36.610.10590.381691.0

Findings.

  1. Each form of sharing helps. PDMS increases monotonically from A to D: 87.5 → 88.8 → 89.2 → 90.1 → 91.0. gFVD and depth AbsRel also improve at every step.
  2. Cross-modal interaction is isolated cleanly. C-mask and C have identical parameters and query allocation. Enabling cross-modal query interaction alone improves PDMS by 0.9 and gFVD from 8.92 to 7.35. This change is larger than the seed standard deviation of PDMS (0.15 over three training seeds; Appendix G.4).
  3. Shared queries add further gains. At the same parameter count, D improves over C in gFVD (7.35 → 6.61), depth AbsRel (0.1180 → 0.1059), and PDMS (90.1 → 91.0). Shared visual queries and the unified spatial reference are introduced together, so this step evaluates their combined effect rather than each one separately.
  4. The gains are not due to capacity. The joint-encoder variants (C-mask, C, D) have 109M fewer parameters than A, yet perform best.

In summary, every step toward unification improves gFVD, depth error, and PDMS, and the full unified design achieves the best results on all three at the lowest parameter count. In the revised manuscript, these results are reported in Table 3 (Section 4.3), and the variants are defined in Appendix G.1.

AuthorsOpenReview ↗

Response to Reviewer (3/4): Novelty (W3)

The core contribution of this work is a new representation for driving world–action models: what is tokenized, and how, and the evidence that this changes downstream prediction and planning. We deliberately build it from well-established components (query-based attention, temporal compression, flow matching, and GRPO), so that the gains can be attributed to the representation itself. The novel elements are:

  1. One jointly decodable latent for appearance, metric geometry, and ego motion. DriveUniTok compresses front-camera RGB, projected LiDAR depth, and clip-relative ego pose into one latent of 584 × 16 scalars per clip, from which all three are decoded. Ego motion is part of the latent rather than a separate action stream, so DriveUniWAM forecasts scene evolution and ego motion as one sequence with a single denoiser.
  2. Shared visual queries without an anchor modality. The same 576 visual queries aggregate RGB and depth features through sequential deformable cross-attention on a common spatial grid, and interact with 8 ego queries through self-attention. Table 3 shows this outperforms both partitioned queries and partitioned queries with full cross-modal attention.
  3. Objectives that enforce cross-modal decodability. Intra- and cross-modal masked reconstruction, together with a training-only semantic target, require each modality to be recoverable from the shared latent (ablation in Appendix D, Table 8).

Positioning against prior representations (summarizing Figure 1 and Section 2; ✓ = inside the decodable latent, sep. = separate stream, ✗ = not modeled):

RepresentationAppearanceMetric geometryEgo motionHow actions are produced
Video VAEs: Wan, HunyuanVideo, Cosmos [1–3]✓✗✗–
BEV tokenizers: BEV-VAE, DriveTok [4, 5]✓scene-level✗–
BEVWorld [6] (RGB + LiDAR BEV latent)✓✓✗–
Occupancy tokenizers: OccWorld, I²-World [7, 8]✗✓ (occupancy)sep.separate ego stream
Any-to-any: 4M-21 [9]✓✓✗modality-specific tokenizers
Latent-action: villa-X, XR-1 [10, 11]✓✗aligned latent actionseparate streams aligned via an anchor
WAMs: Epona [12], PWM [13], DrivingGPT [14], DriveLaW [15], DriveWAM [16]✓ (RGB latent)✗sep.action head / interleaved action tokens / action model on video features
DriveUniTok + DriveUniWAM (Ours)✓✓✓decoded from the same predicted latent

To our knowledge, DriveUniTok is the first driving tokenizer that places appearance, metric geometry, and ego motion in a single jointly decodable latent and uses it directly as the state space of a world–action model.

Evidence. Under matched representation budgets, the unified design improves gFVD from 12.80 to 6.61 and PDMS from 87.5 to 91.0, with fewer tokenizer parameters (847.3M vs. 956.6M; Table 2). The controlled ablations in Response 2/4 (Table 3) show that the gains come from the unified query design beyond shared decoding and cross-modal attention. The benefit also holds across five video diffusion backbones (Section 4.5, Appendix G.7) and through RL post-training (Appendix G.4).

[1] Team Wan et al., Wan, arXiv 2025. [2] Kong et al., HunyuanVideo, arXiv 2024. [3] Agarwal et al., Cosmos, arXiv 2025. [4] Chen & Zhao, BEV-VAE, arXiv 2025. [5] Zhuo et al., DriveTok, arXiv 2026. [6] Zhang et al., BEVWorld, arXiv 2024. [7] Zheng et al., OccWorld, ECCV 2024. [8] Liao et al., I²-World, arXiv 2025. [9] Bachmann et al., 4M-21, NeurIPS 2024. [10] Chen et al., villa-X, arXiv 2025. [11] Fan et al., XR-1, arXiv 2025. [12] Zhang et al., Epona, ICCV 2025. [13] Zhao et al., PWM, NeurIPS 2025. [14] Chen et al., DrivingGPT, ICCV 2025. [15] Xia et al., DriveLaW, CVPR 2026. [16] Shi et al., DriveWAM, arXiv 2026.

Comments on the paper

6 comments
AuthorsOpenReview ↗

Response to Reviewer (1/2): Reconstruction metrics decoupled from training (W1, Q1) and isolating unification (W2)

We thank the reviewer for the careful reading and for recognizing the controlled design of our comparisons.

W1 / Q1. Reconstruction quality under metrics decoupled from the training signal

(a) Backbone capacity and DINO-aligned objectives are held fixed in our controlled comparison. All unified and separate variants in Tables 2, 3, and 10 use the same frozen DINOv3-L features and the same training objectives, including the DINOv3 perceptual and semantic losses. Any advantage of Unified over these variants therefore cannot come from the backbone or from objective–metric alignment. We further evaluate with metrics that the tokenizer is not trained on:

Tokenizer (same DINOv3-L backbone and objectives)DreamSim ↓SigLIP similarity ↑Depth AbsRel ↓
Separate (Equal Budget)0.087580.913140.0766
Separate (Natural Budget)0.079820.927740.0723
Separate Enc. + Shared Dec.0.077290.942100.0679
Unified (Ours)0.076370.948030.0461

Unified is best on all three metrics. SigLIP [2] is a language-supervised image encoder that is entirely independent of our training signal. DreamSim [1] includes DINO-family features among its ensemble components, so we report it together with SigLIP rather than on its own. The depth reconstruction error, which has no relation to DINO, improves by 40% over the equal-budget control.

(b) Pixel-level and I3D metrics in Table 1 reflect a much higher compression rate. The gap to large video VAEs is mainly due to compression. For a 5-frame 288 × 512 clip, Wan2.1 and HunyuanVideo VAEs (4×8×8, width 16) use about 73.7K latent scalars for RGB only. DriveUniTok uses 9,344 scalars for RGB, depth, and ego pose together, about 8× fewer. At this budget, high-frequency texture is the first information to be discarded. PSNR, SSIM, and I3D-FVD are most sensitive to texture, whereas metrics such as V-JEPA FVD and LPIPS weight scene structure more heavily. Table 1 accordingly characterizes the fidelity–compression trade-off under each tokenizer's native configuration.

(c) The latent is a faithful decodable interface for world–action modeling. By this we refer to retaining the appearance, geometry, and ego-motion information that matters for prediction and planning, with all three decodable from the latent, rather than pixel-exact reconstruction. Two results support this:

  • The latent is a better interface than higher-fidelity VAEs. With the video backbone fixed, replacing the native VAE with DriveUniTok improves PDMS by 2.5 points for Wan2.1-1.3B and by 3.1 points for Wan2.2-5B, even though the native VAEs have better PSNR (Section 4.5, Table 16).
  • Geometry and motion are decoded accurately. The depth AbsRel is 0.0461 and the ego ADE is 0.0728 (Table 2).

The additional metrics are reported in Table 10 and Appendix G.3.

W2. The unified-vs-separate comparison does not isolate unification

To isolate these factors, we have added three intermediate controls under the same 584-token budget, backbone, objectives, and training protocol:

VariantgFVD ↓AbsRel ↓ADE ↓PDMS ↑
A. Separate (Equal Budget)12.800.18500.418687.5
B. Separate Enc. + Shared Dec. (cross-modal latent access, shared ego readout)9.450.14200.395088.8
C-mask. Joint Enc., partitioned queries, no cross-group attention8.920.13850.402089.2
C. Same as C-mask, attention mask removed7.350.11800.365090.1
D. Unified: shared visual queries on a common spatial grid6.610.10590.381691.0
  • Parameter sharing and decoder connectivity (A → B) explain only part of the gain (+1.3 PDMS).
  • Cross-modal attention alone (C-mask → C), with identical parameters and query allocation, adds +0.9 PDMS, above the seed standard deviation of 0.15 (Appendix G.4).
  • Shared queries (C → D), the core of the unified latent, add a further +0.9 PDMS and the best gFVD and depth error at the same parameter count. This step introduces shared queries and the common spatial reference together, so it measures their combined effect.

Overall, parameter sharing and cross-modal attention each account for only part of the improvement, and the unified latent design adds a clear further gain on top of both.

In the revised manuscript, these controls are reported in Table 3 and Appendix G.1. The RL comparison on the separate representation is addressed in Response 2/2.

[1] Fu et al., DreamSim, NeurIPS 2023. [2] Zhai et al., SigLIP, ICCV 2023.

AuthorsOpenReview ↗

Response to Reviewer (2/2): RL on separate representations (W2), inference cost (Q2), and camera-only construction (Q3)

W2 (continued). RL post-training on separate representations

We applied the same Flow-GRPO procedure to the equal-budget separate-stream representation, with identical RL configuration, training budget, and test-time settings. Both tokenizers stay frozen.

RepresentationSFT PDMSRL PDMSΔPDMS
Separate (Equal Budget)87.5087.70+0.20
Unified (Ours)91.0091.54+0.54

The unified advantage exists before RL (3.50 points) and is preserved after RL (3.84 points). RL therefore does not close the gap left by the separate representation. On the unified representation, the RL gain is reproducible: across three paired training seeds it is +0.58 ± 0.20, positive in every seed (Appendix G.4, Table 12). These results show that the planning advantage of the unified representation holds both with and without RL post-training. They are reported in Appendix G.4 (Table 11).

Q2. Inference cost per planning decision and real-time feasibility

All models use 10 Euler denoising steps and a single joint sample, with no candidate reranking (Appendix C.3). One planning decision includes historical encoding, 10-step denoising, and ego-only decoding. It costs 172.12 TFLOPs, of which 163.29 TFLOPs is the DiT.

Measured wall-clock latency (batch size 1, from preprocessed CPU tensors to the returned trajectory):

StageH20, SFT (mean / P95 ms)H20, RL (mean / P95 ms)
Historical encoding132.95 / 140.03132.13 / 132.93
10-step DiT denoising1661.28 / 1696.221737.31 / 1767.53
Ego trajectory decoding62.05 / 62.2061.54 / 61.92
End-to-end (incl. I/O and post-processing)1907.63 / 1940.731986.92 / 2012.37

On a single RTX 4090, the same SFT configuration takes 1,363 ms per decision. The 4090 is faster than the H20 because the H20 has lower dense compute throughput. For context, reported latencies of other video-generative WAMs are of the same order: DriveWAM 1,262 ms (H20, per chunk), DriveLaW 1,167 ms, Metis with video 1,380 ms, and PWM with video 830 ms (RTX 4090); see Table 15.

Driving-frame-rate deployment. The current configuration runs at about 0.7 Hz on a single RTX 4090. Reaching the 2 Hz planning rate of NAVSIM requires roughly a 3× speed-up, and the latency profile makes this path clear. Denoising accounts for about 87% of the latency, while encoding and ego decoding together take under 200 ms. Based on the measured per-step cost, a 2-step sampler would take about 0.41 s per decision on an RTX 4090 (≈2.4 Hz). Such few-step samplers are routinely obtained for flow-matching models through step distillation or consistency-style training. Because planning needs only the ego trajectory, RGB and depth decoding is already skipped at inference. The tokenizer and the unified latent are independent of the sampler, so such acceleration is complementary to our contribution. We discuss these deployment considerations in Appendix G.5.

Q3. Construction of the camera-only variant

The camera-only model is trained independently, not obtained by masking LiDAR at evaluation time. Its tokenizer is trained separately with the same 584 × 16 latent budget, using front-camera RGB and relative ego poses as input. Its DriveUniWAM starts from the same Wan2.2-TI2V-5B initialization with an identical SFT configuration, and the architecture, data split, and optimization schedule all match the full model.

Under this matched comparison, camera-only reaches 90.5 PDMS versus 91.0 for the full model. LiDAR geometry therefore contributes about 0.5 PDMS, and the camera-only variant is already above the single-camera DriveWAM (90.1). Test-time masking of the full model without retraining is a separate experiment, reported in Appendix F (Table 9). The construction is described in Appendix G.6.

AuthorsOpenReview ↗

Revised Manuscript Uploaded

Dear Reviewers and Area Chair,

Thank you again for your thoughtful comments and constructive suggestions. We have uploaded a revised manuscript incorporating the additional experiments and clarifications discussed in our responses. For your convenience, the revised and newly added content is highlighted in blue throughout the manuscript, making the changes easier to locate. The main updates include: (1) additional architectural ablations and cross-modal consistency evaluations; (2) further reconstruction and RL comparisons; (3) clarifications on inference settings, computational costs, latency, and multi-seed results; and (4) a more precise discussion of our contributions and limitations.

Your feedback has been very helpful in strengthening our work. We sincerely appreciate the time and care you have devoted to reviewing our paper, and hope the revised manuscript addresses your concerns. Thank you again for your consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.