- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
DriveUniTok encodes front-camera RGB, projected LiDAR depth, and clip-relative ego pose into one continuous latent of 584 tokens with width 16 per five-frame clip. Shared queries aggregate the modalities, while modality-specific readouts reconstruct them. The frozen tokenizer is coupled to a pretrained video diffusion backbone for future-latent prediction and then refined with Flow-GRPO for planning. The authors report 91.0 supervised PDMS, 91.54 after reinforcement learning, and 90.3 EPDMS on NAVSIM-v2.
Critical strengths
S1. The shared bottleneck addresses a relevant representation question: whether visual, geometric, and ego-motion signals can be jointly encoded before driving-world prediction.
S2. Table 2 provides natural- and equal-budget controls. At equal downstream cost, PDMS rises from 87.5 to 91.0, with lower future prediction errors. Relative to the natural-budget control, per-step DiT computation falls from 34.2 to 16.3 TFLOPs. Results across five backbones and on NAVSIM-v2 add useful breadth.
Critical weaknesses
W1. The comparison changes more than latent organization. Parameter sharing, cross-modal connectivity, decoder visibility, and ego readout also differ (Section 4.3; Appendix A). Table 2 therefore supports the complete design, but does not isolate the benefit of joint tokenization.
W2. The claimed cross-modal consistency is not measured directly. LPIPS/FVD, depth error, and trajectory error assess outputs separately; they do not show whether modalities agree within the same predicted future. The language in Sections 3.2–3.3 should be qualified or supported by a direct consistency measure.
W3. Test-time planning settings are insufficiently specified. Appendix C.2 describes sampling during RL training, but the denoising steps, candidate count, trajectory selection rule, and any additional scorer used for evaluation remain unclear. This limits reproducibility and makes the computational comparison hard to interpret.
W4. The smaller planning gains lack uncertainty estimates. The 0.54 PDMS increase after RL coincides with lower NC (99.0 to 98.7) and TTC (96.3 to 95.6) in Table 3. Seed variation or paired intervals would clarify whether these changes are reliable and how to interpret the safety-metric trade-off.
Questions for authors
Q1. Can the authors compare against separate encoders with unrestricted shared decoding, and report a direct geometric or motion-consistency measure for generated futures?
Q2. What are the test-time denoising budget, candidate count, and trajectory selection procedure for each main table? Do the supervised and RL models use identical settings?
Q3. How much do the principal planning results vary across seeds or paired evaluation runs? Are the NC and TTC decreases after RL consistent?
4 comments
Response to Reviewer (2/2): Test-time protocol (W3, Q2) and seed robustness (W4, Q3)
W3 / Q2. Test-time denoising budget, candidates, and trajectory selection
The test-time protocol is identical for every model in the main prediction and planning tables (Tables 2–6), including all architectural variants:
| Setting | Value |
|---|---|
| Sampler | 10 deterministic Euler steps, one DiT forward pass per step |
| Candidates per scene | 1 (a single joint sample containing two future clips) |
| Trajectory selection | None: no candidate reranking, no additional inference-time scorer |
| Output | Decoded clip-relative poses composed into 8 waypoints at 0.5 s over 4 s, submitted directly to NAVSIM |
| SFT vs. RL | Identical settings |
The G = 8 stochastic Flow-SDE samples in Appendix C.2 are used only during RL training to compute group-relative advantages. They are never used for test-time selection, and PDMS is computed only for evaluation. The SFT → RL gain is therefore measured at the same single-sample inference budget.
This also makes the computational comparison straightforward to interpret. One planning decision costs 172.12 TFLOPs, of which 163.29 TFLOPs is the 10-step DiT. Measured end to end, it takes 1,908 ms on an H20 and 1,363 ms on an RTX 4090. The SFT and RL models have the same cost per decision.
In the revised manuscript, these settings are stated in Section 4.1 and Appendix C.3, the RL training details are in Appendix C.2, and the full cost and latency breakdown is in Appendix G.5.
W4 / Q3. Seed variation and paired evaluation
We trained three independent SFT runs (seeds 0, 1, 42) and initialized each RL run from its matched SFT checkpoint. The frozen tokenizer and evaluation settings were shared, and every checkpoint was evaluated on the full NAVSIM navtest set.
| Metric | SFT (mean ± std) | SFT + RL (mean ± std) | Paired Δ (RL − SFT) | Per-seed Δ |
|---|---|---|---|---|
| PDMS ↑ | 90.97 ± 0.15 | 91.55 ± 0.05 | +0.58 ± 0.20 | +0.80 / +0.40 / +0.54 |
| EP ↑ | 85.57 ± 0.35 | 87.37 ± 0.25 | +1.80 ± 0.10 | – |
| NC ↑ | 99.00 ± 0.10 | 98.70 ± 0.10 | −0.30 ± 0.20 | −0.50 / −0.10 / −0.30 |
| TTC ↑ | 96.30 ± 0.20 | 95.60 ± 0.20 | −0.70 ± 0.00 | −0.70 in each run |
Entries are mean ± sample standard deviation over three seeds; paired differences are computed within each seed.
- The PDMS gain is reliable. RL improves PDMS in every seed (+0.80 / +0.40 / +0.54). The mean gain of +0.58 is nearly 4× the seed-to-seed standard deviation of the SFT model (0.15), and the RL models are even more stable across seeds (std 0.05). The improvement therefore clearly exceeds training variance.
- Interpreting the NC/TTC change: RL makes the policy less conservative. The NC and TTC decreases occur in every seed, so they reflect a systematic change in driving behavior rather than noise. The supervised model is conservative: its ego progress (85.6) is below the human reference (87.5). Our RL reward follows the PDM score form with the NC and DAC gates intact, and, following ReCogDrive, gives ego progress twice its evaluation weight (EP : TTC : Comfort = 10 : 5 : 2; Appendix C.2). RL post-training therefore shifts the policy toward more assertive driving. Ego progress rises to 87.4, essentially matching the human level, at the cost of slightly smaller safety margins. Since TTC flags any predicted time-to-collision below a fixed threshold, closer following and earlier merging are the first behaviors to register there. This is the expected outcome of optimizing a composite planning reward, and the net effect is positive, with PDMS improving in all three seeds. In absolute terms the safety metrics stay high: NC remains at 98.7, comparable to the strongest planners in Table 5.
In the revised manuscript, these results are in Appendix G.4 (Table 12), and the matched RL control for the separate representation is in Table 11.
Summary of our responses to Reviewer AUSH
Dear Reviewer AUSH,
Thank you again for your precise and constructive questions. Below is a brief map from each point to our response and the revised manuscript (changes in blue):
- W1 / Q1 (separate encoders with unrestricted shared decoding): Added exactly this control (B), plus two joint-encoder controls. B reaches 88.8 PDMS vs. 91.0 for Unified, so decoder-level sharing does not explain the gain. → Response 1/2; Table 3, Appendix G.1.
- W2 / Q1 (direct consistency measure): Depth–ego reprojection consistency and RGB–depth–ego rigid-flow EPE on generated futures, with errors reduced by 47.5% / 42.3%. The claims in the abstract and Section 3.2 are now qualified to static-region geometric consistency. → Response 1/2; Table 4, Appendix G.2.
- W3 / Q2 (test-time protocol): 10 Euler steps, a single sample, no reranking or extra scorer, identical for SFT and RL. G = 8 samples are used only during RL training. → Response 2/2; Section 4.1, Appendix C.3.
- W4 / Q3 (seed variation, NC/TTC): Three paired seeds give a PDMS gain of +0.58 ± 0.20, positive in every seed. The consistent NC/TTC decrease reflects a less conservative policy, with ego progress rising to human level under a progress-weighted reward (Appendix C.2). → Response 2/2; Appendix G.4.
We would be grateful to know whether these address your concerns, or if any point needs further clarification.
Follow-up on our responses
Dear Reviewer AUSH,
Thank you again for your precise and constructive questions. We have added every experiment you requested: the separate-encoder / shared-decoder control, direct consistency measures for generated futures, the full test-time protocol, and three-seed paired results with an analysis of the NC/TTC change (Responses 1/2–2/2, summarized above).
As the discussion period is drawing to a close, we would greatly appreciate your feedback. If our responses have resolved your concerns, we would be sincerely grateful if you could reconsider your score in light of the new results. If any point remains unclear, we would be glad to clarify further.
Best regards,
The Authors
Response to Reviewer (1/2): Architectural controls (W1, Q1) and cross-modal consistency (W2, Q1)
We thank the reviewer for the constructive and precise suggestions. Both requested experiments have been added to the revised manuscript.
W1 / Q1a. Separate encoders with unrestricted shared decoding, and further controls
Table 2 compares the complete designs. To attribute the gain to specific factors (latent organization, parameter sharing, cross-modal connectivity, decoder visibility, and ego readout), we add the exact control the reviewer requested (B), together with two joint-encoder controls. All variants use the same 584-token budget, latent width 16, frozen DINOv3-L features, objectives, DriveUniWAM backbone, and inference settings:
W2 / Q1b. Direct cross-modal consistency of generated futures
Following the reviewer's suggestion, we measure whether the modalities agree within the same predicted future, using each model's own generated RGB, depth, and ego motion:
All methods are scored on the same set of valid static pixels on NAVSIM navtest, defined by a GT-based static and visibility mask intersected with every method's validity mask.
Unified reduces the two errors by 47.5% / 42.3% relative to equal budget, and by 45.6% / 41.6% relative to the natural budget, which uses about twice the downstream tokens. Its geometric consistency is close to the GT reference.
Qualified claims. As the reviewer suggested, we have revised the language. The abstract and Section 3.2 now describe the benefit specifically as static-region geometric consistency and point to the direct measurement.
In the revised manuscript, the controls are reported in Table 3 and Appendix G.1, and the consistency results and protocol in Table 4 and Appendix G.2.
[1] Bian et al., SC-SfMLearner, NeurIPS 2019. [2] Yin & Shi, GeoNet, CVPR 2018. [3] Teed & Deng, RAFT, ECCV 2020.