- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes DriveTok, a unified 3D driving scene tokenizer designed to encode multi-view camera observations into a shared scene representation for reconstruction and understanding tasks. The proposed framework first extracts visual features using a pretrained DINOv3 backbone and aggregates multi-camera information into BEV-aligned scene tokens through deformable cross-attention. A spatial-aware multi-view transformer is then introduced to facilitate interactions between scene tokens and camera-specific view tokens, leveraging Plücker ray embeddings and visibility-guided attention. The resulting representations are jointly optimized for RGB reconstruction, depth estimation, semantic segmentation, 3D occupancy prediction, and latent semantic regularization. Experiments on nuScenes demonstrate competitive performance across multiple tasks.
Critical strengths
-
The paper addresses the challenge of integrating multi-camera observations into a unified representation that supports both low-level reconstruction and high-level scene understanding.
-
Comprehensive multi-task formulation. The framework combines image reconstruction, depth estimation, semantic understanding, and 3D occupancy prediction within a single architecture. This is an appealing direction for autonomous driving representations.
-
Promising empirical results. DriveTok achieves competitive occupancy prediction performance and strong depth estimation results on nuScenes. The evaluation also includes several useful ablations and experiments on downstream forecasting and planning tasks.
Critical weaknesses
- Insufficient differentiation from existing approaches
The primary concern is that DriveTok largely integrates well-established architectural components. Specifically, the proposed scene encoder adopts BEVFormer-style deformable attention to aggregate multi-view features into a unified BEV representation. The decoder employs standard transformer operations combined with Plücker ray embeddings and visibility-guided attention. Furthermore, the multi-task training strategy follows the conventional paradigm of jointly optimizing reconstruction and perception objectives. While this combination is reasonable, the technical contributions appear incremental.
- The baselines compared in this work are mostly 2025 methods, and comparisons against the state-of-the-art approaches are missing, including:
[1] GenieDrive: Towards Physics-Aware Driving World Model with 4D Occupancy Guided Video Generation
[2] SparseWorld-TC: Trajectory-Conditioned Sparse Occupancy World Model
[3] DrivingRecon: Large 4D Gaussian Reconstruction Model For Autonomous Driving
[4] Doe-2: 3D Representation World Model for Unified Driving Scene Forecasting
[5] Streaming Dense Voxel Representations for 3D Occupancy Prediction
- Limited evidence of cross-view consistency and generalization
The paper repeatedly emphasizes that DriveTok produces spatially consistent, camera-agnostic, and resolution-independent scene tokens. However, these properties are not directly evaluated in sufficient detail. For cross-view consistency, the evidence mainly consists of qualitative reconstruction and semantic prediction results. There are no quantitative measurements of feature consistency or geometric agreement in overlapping camera regions.
- Downstream forecasting and planning results lack sufficient methodological details
Section 4.4 introduces DriveTok-World and reports substantial improvements in 4D occupancy forecasting and motion planning. However, the paper provides very limited information about the architecture and training procedure of this downstream model. In particular, it is unclear how the scene tokens are incorporated into the temporal forecasting framework, which predictive modules are used, and whether the baselines share comparable supervision, architectures, and training configurations. This issue is especially important given the substantial improvements reported in Tabel 5. DriveTok-World achieves 60.73 average geometric IoU compared with 44.07 for the strongest listed baseline. Without a controlled experimental setup, it is difficult to determine whether these improvements originate from the tokenizer itself or from other differences in the downstream forecasting pipeline.
Questions for authors
Please refer to the above weaknesses and these questions.
Q1: Provide the more comparisons with the state-of-the-art approaches.
Q2: Provide more details about the architecture and training procedure of DriveTok-World, particularly how scene tokens are incorporated into the temporal forecasting framework. Given the substantial improvement in Table 5, could controlled experiments with identical downstream architectures, supervision, and training configurations be conducted to isolate the contribution of the proposed tokenizer?
Q3: How is the claimed cross-view consistency of the learned scene tokens quantitatively validated? Can supplementary experiments quantify geometric agreement or feature consistency across overlapping camera views?
9 comments
Author Response - Part 2
W2 & Q1: Quantitative Comparisons and Benchmark Categorization Across Autonomous Driving Tasks
Recent representative studies (DrivingRecon, StreamOcc, GenieDrive, SparseWorld-TC, and Doe-2) target distinct problem settings and benchmark tracks within the autonomous driving literature. To ensure fair and meaningful comparisons, we report results aligned with the standardized benchmarks for each task:
1. Benchmark 1: nuScenes Current-Frame 3D Semantic Occupancy Benchmark
For current-timestamp 3D environmental occupancy perception, comparative evaluations on the nuScenes SurroundOcc Benchmark are summarized below:
Table 2: Current-timestamp 3D occupancy benchmark evaluation on the nuScenes validation set
| Method | Occupancy IoU (%) ↑ | Occupancy mIoU (%) ↑ |
|---|---|---|
| SurroundOcc | 31.49 | 20.30 |
| GaussianFormer-2 | 30.56 | 20.02 |
| VG3S | 34.41 | 21.52 |
| StreamOcc | 33.80 | 23.40 |
| DriveTok (Ours) | 33.32 | 20.06 |
Task Scope Clarification: VG3S and StreamOcc primarily target 3D semantic occupancy prediction. DriveTok unifies reconstruction and understanding, encoding single-timestamp multi-view images into shared scene tokens that support RGB reconstruction, metric depth estimation, semantic prediction, and 3D occupancy prediction. Under this joint-task setting, DriveTok achieves 33.32% IoU and 20.06% mIoU. We claim comparable occupancy performance, with the goal of preserving appearance, geometry, and semantics in a unified scene representation.
2. Benchmark 2: nuScenes Future 4D Occupancy Forecasting Benchmark
For spatiotemporal rollout tasks targeted by world models, evaluations on the standardized nuScenes 4D Occupancy Forecasting Benchmark (predicting future 1s / 2s / 3s states given historical observations) are detailed below:
Table 3: Performance comparison on the nuScenes future 4D occupancy forecasting benchmark
| Method | Mean Future Semantic mIoU (%) ↑ | Mean Future Geometric IoU (%) ↑ |
|---|---|---|
| DOME-F | 18.25 | 28.84 |
| I2-World-STC | 18.97 | 28.77 |
| DTT | 19.60 | 36.11 |
| COME-F | 22.26 | 44.07 |
| DriveTok-World (Ours) | 26.29 (+2.41) | 60.73 (+13.80) |
Benchmark Protocol Clarification: On the nuScenes future 4D occupancy forecasting benchmark, DriveTok-World achieves 26.29% mean semantic mIoU and 60.73% mean geometric IoU, outperforming all methods listed in Table 3. Compared with COME-F, these correspond to improvements of 4.03 and 16.66 percentage points, respectively.
3. Task Scope Clarification: Reconstruction, Video Generation, and Planning
- DrivingRecon: Designed for offline high-fidelity novel-view synthesis, evaluated primarily via image rendering metrics (PSNR, SSIM, and LPIPS) on Waymo and nuScenes, without closed-loop planning or discrete decision-making evaluations. Using a compact budget of 1,024 scene tokens, DriveTok achieves 27.89 dB PSNR and 0.747 SSIM on nuScenes while supporting joint occupancy and depth prediction.
- GenieDrive: Formulated as a trajectory-conditioned long-video generation model, evaluated primarily on video synthesis fidelity (FVD and FID) without generating vehicle control actions or planning trajectories.
- System Scope Summary: These approaches focus on distinct benchmark tracks and do not target end-to-end planning metrics. DriveTok serves as a unified 3D scene tokenizer, providing a compact, geometrically consistent representation that can naturally serve as structured conditioning for downstream world models (e.g., DriveTok-World) or video synthesis pipelines.
Author Response - Part 3
W3 & Q3: Quantitative Evaluation and Ablation of Cross-View Physical Consistency
To address the evaluation of cross-view geometric consistency, we established a physical evaluation protocol based on LiDAR point projections across all 6,019 nuScenes validation frames:
- Evaluation Protocol and Dataset Scale: We extracted physical LiDAR points co-observed across adjacent overlapping camera frustums, yielding 1,169,768 physical correspondence pairs (1,164,033 pairs containing valid ground-truth LiDARSeg labels).
- Metric Error and Semantic Agreement: Predicted 3D coordinates back-projected from adjacent camera views were transformed into the unified ego coordinate frame to compute absolute Euclidean distance deviations and cross-view semantic agreement at corresponding pixel locations.
Table 4: Quantitative evaluation of cross-view 3D physical geometry and semantic consistency on the nuScenes validation set
| Metric Category | Statistical Metric | Value | Metric Definition and Physical Significance |
|---|---|---|---|
| 3D Physical Geometry Error | Mean 3D Error ↓ | 1.650 m | Mean absolute Euclidean distance of back-projected points in ego frame |
| RMS Error ↓ | 4.004 m | Influenced by large-parallax outliers at distant ranges (40–50 m) | |
| Points w/ Error ≤ 1.0 m (%) ↑ | 67.51% | Proportion of correspondences maintaining error within 1.0 m | |
| Cross-View Semantic Consistency | Exact Semantic Match Rate (%) ↑ | 88.38% | Agreement rate of predicted classes between adjacent views |
| Class-Balanced Semantic Match Rate (%) ↑ | 78.51% | Semantic agreement accounting for background class imbalance | |
| Balanced Both-Correct Accuracy (%) ↑ | 77.80% | Accuracy where both views correctly predict ground-truth LiDARSeg | |
| Class-Balanced Both-Correct Accuracy (%) ↑ | 56.58% | Dual-view prediction accuracy balanced across long-tail categories | |
| Metric Depth Benchmark | AbsRel ↓ / δ < 1.25 ↑ | 0.09627 / 0.89150 | Depth accuracy at projected physical correspondence locations |
Table 5: Inference intervention ablation on decoding geometric mechanisms using identical model weights
| Inference Evaluation Condition | Mean 3D Error (m) ↓ | Points w/ Error ≤ 1m (%) ↑ | Balanced Both-Correct Acc (%) ↑ | Depth AbsRel ↓ | RGB Recon PSNR ↑ |
|---|---|---|---|---|---|
| DriveTok Nominal Model | 1.650 | 67.51 | 56.58 | 0.09627 | 27.376 |
| Intervention: Disable visibility mask | 2.483 | 54.58 | 26.89 | 0.28167 | 19.407 |
| Intervention: Zero-out Plücker ray embeddings | 2.047 | 58.84 | 49.68 | 0.10460 | 26.385 |
Analysis of Cross-View Consistency
- Near-Field Precision and Far-Field Parallax Effects: Under real-world LiDAR projection, 67.51% of correspondence pairs maintain spatial deviations below 1.0 m, with depth AbsRel reaching 0.09627 in overlap zones. The difference between mean error (1.650 m) and RMS error (4.004 m) reflects sensor geometry: at distant ranges (40–50 m), baseline parallax and sparse point returns amplify small pixel-level depth errors upon 3D back-projection, forming an error tail.
- Cross-Camera Semantic Consensus: The cross-view semantic agreement reaches 88.38% (78.51% class-balanced), and the dual-view ground-truth accuracy reaches 77.80%. This indicates that the tokenizer assigns consistent semantic classifications to physical targets regardless of whether they appear in front or side camera frustums.
- Causal Mechanistic Validation: As shown in Table 5, disabling the visibility mask at inference time without retraining degrades cross-view 3D spatial error by +50.5% (rising to 2.483 m), while balanced accuracy drops from 56.58% to 26.89%. This confirms that cross-view consistency stems from physical ray queries and visibility constraints rather than shortcut fitting in 2D feature layers.
Author Response - Part 1
We sincerely thank the reviewer for the in-depth review and constructive guidance regarding the core theoretical formulation, benchmark scopes, cross-view physical consistency validation, and downstream world model architectural mechanisms. Below, we provide technical explanations and empirical results addressing each concern point by point.
W1: Representation Dynamics and Core Technical Contributions
Compressing multi-view autonomous driving cameras into a unified tokenizer that simultaneously preserves fine-grained texture reconstruction and maintains 3D metric spatial topology requires resolving the representation conflict between low-level pixel memorization and global rigid geometry:
1. Theoretical Challenges and Architectural Formulations
Table 1: Systematic ablation on multi-task objectives, geometric priors, and encoding architectures on the nuScenes validation set
2. Key Result Analysis and Architectural Advantages