acceptodds
Under review as a conference paper at ICLR 2027

DriveTok: 3D Driving Scene Tokenization for Unified Multi-View Reconstruction and Understanding

Abstract

Visual tokenization for autonomous driving must represent multi-view scenes compactly while preserving the information needed for reconstruction and scene understanding. However, most existing tokenizers are designed for monocular and 2D scenes, leading to inefficiency and inter-view inconsistency when applied to high-resolution multi-view driving scenes. To address this, we propose DriveTok, an efficient 3D driving scene tokenizer for unified multi-view reconstruction and understanding. DriveTok first obtains semantically rich visual features from vision foundation models and then transforms them into the scene tokens with 3D deformable cross-attention. For decoding, we employ a multi-view transformer to reconstruct multi-view features from the scene tokens and use multiple heads to obtain RGB, depth, and semantic reconstructions. We also add a 3D head directly on the scene tokens for 3D semantic occupancy prediction for better spatial awareness. With the multiple training objectives, DriveTok learns unified scene tokens that integrate semantic, geometric, and textural information for efficient multi-view tokenization. Extensive experiments demonstrate that the scene tokens from DriveTok perform well on image reconstruction, semantic segmentation, depth prediction, and 3D occupancy prediction tasks.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Autonomous Driving, 3D Scene Tokenization, Scene Understanding

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_AGe9OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes DriveTok, a unified 3D driving scene tokenizer designed to encode multi-view camera observations into a shared scene representation for reconstruction and understanding tasks. The proposed framework first extracts visual features using a pretrained DINOv3 backbone and aggregates multi-camera information into BEV-aligned scene tokens through deformable cross-attention. A spatial-aware multi-view transformer is then introduced to facilitate interactions between scene tokens and camera-specific view tokens, leveraging Plücker ray embeddings and visibility-guided attention. The resulting representations are jointly optimized for RGB reconstruction, depth estimation, semantic segmentation, 3D occupancy prediction, and latent semantic regularization. Experiments on nuScenes demonstrate competitive performance across multiple tasks.

Critical strengths
  1. The paper addresses the challenge of integrating multi-camera observations into a unified representation that supports both low-level reconstruction and high-level scene understanding.

  2. Comprehensive multi-task formulation. The framework combines image reconstruction, depth estimation, semantic understanding, and 3D occupancy prediction within a single architecture. This is an appealing direction for autonomous driving representations.

  3. Promising empirical results. DriveTok achieves competitive occupancy prediction performance and strong depth estimation results on nuScenes. The evaluation also includes several useful ablations and experiments on downstream forecasting and planning tasks.

Critical weaknesses
  1. Insufficient differentiation from existing approaches

The primary concern is that DriveTok largely integrates well-established architectural components. Specifically, the proposed scene encoder adopts BEVFormer-style deformable attention to aggregate multi-view features into a unified BEV representation. The decoder employs standard transformer operations combined with Plücker ray embeddings and visibility-guided attention. Furthermore, the multi-task training strategy follows the conventional paradigm of jointly optimizing reconstruction and perception objectives. While this combination is reasonable, the technical contributions appear incremental.

  1. The baselines compared in this work are mostly 2025 methods, and comparisons against the state-of-the-art approaches are missing, including:

[1] GenieDrive: Towards Physics-Aware Driving World Model with 4D Occupancy Guided Video Generation

[2] SparseWorld-TC: Trajectory-Conditioned Sparse Occupancy World Model

[3] DrivingRecon: Large 4D Gaussian Reconstruction Model For Autonomous Driving

[4] Doe-2: 3D Representation World Model for Unified Driving Scene Forecasting

[5] Streaming Dense Voxel Representations for 3D Occupancy Prediction

  1. Limited evidence of cross-view consistency and generalization

The paper repeatedly emphasizes that DriveTok produces spatially consistent, camera-agnostic, and resolution-independent scene tokens. However, these properties are not directly evaluated in sufficient detail. For cross-view consistency, the evidence mainly consists of qualitative reconstruction and semantic prediction results. There are no quantitative measurements of feature consistency or geometric agreement in overlapping camera regions.

  1. Downstream forecasting and planning results lack sufficient methodological details

Section 4.4 introduces DriveTok-World and reports substantial improvements in 4D occupancy forecasting and motion planning. However, the paper provides very limited information about the architecture and training procedure of this downstream model. In particular, it is unclear how the scene tokens are incorporated into the temporal forecasting framework, which predictive modules are used, and whether the baselines share comparable supervision, architectures, and training configurations. This issue is especially important given the substantial improvements reported in Tabel 5. DriveTok-World achieves 60.73 average geometric IoU compared with 44.07 for the strongest listed baseline. Without a controlled experimental setup, it is difficult to determine whether these improvements originate from the tokenizer itself or from other differences in the downstream forecasting pipeline.

Questions for authors

Please refer to the above weaknesses and these questions.

Q1: Provide the more comparisons with the state-of-the-art approaches.

Q2: Provide more details about the architecture and training procedure of DriveTok-World, particularly how scene tokens are incorporated into the temporal forecasting framework. Given the substantial improvement in Table 5, could controlled experiments with identical downstream architectures, supervision, and training configurations be conducted to isolate the contribution of the proposed tokenizer?

Q3: How is the claimed cross-view consistency of the learned scene tokens quantitatively validated? Can supplementary experiments quantify geometric agreement or feature consistency across overlapping camera views?

9 comments
AuthorsOpenReview ↗

Author Response - Part 1

We sincerely thank the reviewer for the in-depth review and constructive guidance regarding the core theoretical formulation, benchmark scopes, cross-view physical consistency validation, and downstream world model architectural mechanisms. Below, we provide technical explanations and empirical results addressing each concern point by point.


W1: Representation Dynamics and Core Technical Contributions

Compressing multi-view autonomous driving cameras into a unified tokenizer that simultaneously preserves fine-grained texture reconstruction and maintains 3D metric spatial topology requires resolving the representation conflict between low-level pixel memorization and global rigid geometry:

1. Theoretical Challenges and Architectural Formulations

  • Representation Conflict: 2D pixel-level reconstruction requires preserving localized high-frequency texture details, which easily causes the latent representation to degrade into view-dependent pixel memorization. Conversely, 3D metric occupancy necessitates extracting viewpoint-invariant, low-frequency rigid geometry. Jointly optimizing these competing objectives can trigger representation collapse and geometric degradation.
  • Visibility-Guided Decoupling Mechanism: We introduce a physical visibility attention mask coupled with Plücker ray queries, forcing scene tokens to interact exclusively with physically visible light rays within their corresponding frustums. This design prevents spurious cross-view projection gradients, enabling a single continuous scene representation to balance appearance reconstruction with metric 3D structure.

Table 1: Systematic ablation on multi-task objectives, geometric priors, and encoding architectures on the nuScenes validation set

Evaluation DimensionExperimental SetupPSNR ↑SSIM ↑Occupancy IoU (%) ↑Occupancy mIoU (%) ↑
Training ObjectivesRGB-only supervision (Standard image tokenizer)28.840.77012.813.84
RGB + Depth28.520.76420.729.36
RGB + Depth + Semantics28.060.75624.0511.68
RGB + Depth + Semantics + Occupancy (DriveTok full)27.890.74733.3220.06
Geometric GuidanceW/o physical visibility attention mask (w/o Geo-Attn)29.980.7985.320.59
W/ physical visibility attention mask (w/ Geo-Attn)28.840.77012.813.84
Positional Encoding2D image positional encoding only24.690.63831.9819.39
3D Plücker ray encoding only23.670.60932.2119.47
2D positional encoding + 3D Plücker ray encoding25.240.66732.9619.53
Representation ArchitectureVGGT tokenizer architecture (Wang et al., 2025a)25.450.70325.088.60
DriveTok scene tokenizer27.890.74733.3220.06

2. Key Result Analysis and Architectural Advantages

  • Synergistic multi-task gains: As depth, semantic, and occupancy supervisions are progressively introduced, 3D occupancy mIoU increases substantially from 3.84% to 20.06% (a net improvement of +16.22 percentage points) while reconstruction quality remains high at 27.89 dB. This demonstrates that multi-task supervision guides the representation to encode dense physical geometry without causing representational collapse.
  • Elimination of geometric collapse via visibility constraints: The 0.59% and 3.84% scores in the geometric guidance ablation were measured under a frozen probing protocol following RGB-only pretraining (without any 3D annotations). Under zero geometric supervision, removing the visibility mask causes the latent representation to collapse into 2D pixel-level memorization (yielding near-chance probing accuracy of 0.59%). In contrast, visibility constraints enforce the emergence of coherent 3D manifolds within the frustum (improving accuracy by over 6×6\times to 3.84%), which further scales to 20.06% under full end-to-end training.
  • Metric scene grid vs. view-centric tokens: Under identical task heads and supervision signals, DriveTok outperforms the VGGT baseline by +2.44 dB in PSNR and +11.46 percentage points in occupancy mIoU, demonstrating the advantage of consolidating multi-view evidence into a metric 3D coordinate space.
AuthorsOpenReview ↗

Author Response - Part 2

W2 & Q1: Quantitative Comparisons and Benchmark Categorization Across Autonomous Driving Tasks

Recent representative studies (DrivingRecon, StreamOcc, GenieDrive, SparseWorld-TC, and Doe-2) target distinct problem settings and benchmark tracks within the autonomous driving literature. To ensure fair and meaningful comparisons, we report results aligned with the standardized benchmarks for each task:

1. Benchmark 1: nuScenes Current-Frame 3D Semantic Occupancy Benchmark

For current-timestamp 3D environmental occupancy perception, comparative evaluations on the nuScenes SurroundOcc Benchmark are summarized below:

Table 2: Current-timestamp 3D occupancy benchmark evaluation on the nuScenes validation set

MethodOccupancy IoU (%) ↑Occupancy mIoU (%) ↑
SurroundOcc31.4920.30
GaussianFormer-230.5620.02
VG3S34.4121.52
StreamOcc33.8023.40
DriveTok (Ours)33.3220.06

Task Scope Clarification: VG3S and StreamOcc primarily target 3D semantic occupancy prediction. DriveTok unifies reconstruction and understanding, encoding single-timestamp multi-view images into shared scene tokens that support RGB reconstruction, metric depth estimation, semantic prediction, and 3D occupancy prediction. Under this joint-task setting, DriveTok achieves 33.32% IoU and 20.06% mIoU. We claim comparable occupancy performance, with the goal of preserving appearance, geometry, and semantics in a unified scene representation.

2. Benchmark 2: nuScenes Future 4D Occupancy Forecasting Benchmark

For spatiotemporal rollout tasks targeted by world models, evaluations on the standardized nuScenes 4D Occupancy Forecasting Benchmark (predicting future 1s / 2s / 3s states given historical observations) are detailed below:

Table 3: Performance comparison on the nuScenes future 4D occupancy forecasting benchmark

MethodMean Future Semantic mIoU (%) ↑Mean Future Geometric IoU (%) ↑
DOME-F18.2528.84
I2-World-STC18.9728.77
DTT19.6036.11
COME-F22.2644.07
DriveTok-World (Ours)26.29 (+2.41)60.73 (+13.80)

Benchmark Protocol Clarification: On the nuScenes future 4D occupancy forecasting benchmark, DriveTok-World achieves 26.29% mean semantic mIoU and 60.73% mean geometric IoU, outperforming all methods listed in Table 3. Compared with COME-F, these correspond to improvements of 4.03 and 16.66 percentage points, respectively.

3. Task Scope Clarification: Reconstruction, Video Generation, and Planning

  • DrivingRecon: Designed for offline high-fidelity novel-view synthesis, evaluated primarily via image rendering metrics (PSNR, SSIM, and LPIPS) on Waymo and nuScenes, without closed-loop planning or discrete decision-making evaluations. Using a compact budget of 1,024 scene tokens, DriveTok achieves 27.89 dB PSNR and 0.747 SSIM on nuScenes while supporting joint occupancy and depth prediction.
  • GenieDrive: Formulated as a trajectory-conditioned long-video generation model, evaluated primarily on video synthesis fidelity (FVD and FID) without generating vehicle control actions or planning trajectories.
  • System Scope Summary: These approaches focus on distinct benchmark tracks and do not target end-to-end planning metrics. DriveTok serves as a unified 3D scene tokenizer, providing a compact, geometrically consistent representation that can naturally serve as structured conditioning for downstream world models (e.g., DriveTok-World) or video synthesis pipelines.
AuthorsOpenReview ↗

Author Response - Part 3

W3 & Q3: Quantitative Evaluation and Ablation of Cross-View Physical Consistency

To address the evaluation of cross-view geometric consistency, we established a physical evaluation protocol based on LiDAR point projections across all 6,019 nuScenes validation frames:

  • Evaluation Protocol and Dataset Scale: We extracted physical LiDAR points co-observed across adjacent overlapping camera frustums, yielding 1,169,768 physical correspondence pairs (1,164,033 pairs containing valid ground-truth LiDARSeg labels).
  • Metric Error and Semantic Agreement: Predicted 3D coordinates back-projected from adjacent camera views were transformed into the unified ego coordinate frame to compute absolute Euclidean distance deviations and cross-view semantic agreement at corresponding pixel locations.

Table 4: Quantitative evaluation of cross-view 3D physical geometry and semantic consistency on the nuScenes validation set

Metric CategoryStatistical MetricValueMetric Definition and Physical Significance
3D Physical Geometry ErrorMean 3D Error ↓1.650 mMean absolute Euclidean distance of back-projected points in ego frame
RMS Error ↓4.004 mInfluenced by large-parallax outliers at distant ranges (40–50 m)
Points w/ Error ≤ 1.0 m (%) ↑67.51%Proportion of correspondences maintaining error within 1.0 m
Cross-View Semantic ConsistencyExact Semantic Match Rate (%) ↑88.38%Agreement rate of predicted classes between adjacent views
Class-Balanced Semantic Match Rate (%) ↑78.51%Semantic agreement accounting for background class imbalance
Balanced Both-Correct Accuracy (%) ↑77.80%Accuracy where both views correctly predict ground-truth LiDARSeg
Class-Balanced Both-Correct Accuracy (%) ↑56.58%Dual-view prediction accuracy balanced across long-tail categories
Metric Depth BenchmarkAbsRel ↓ / δ < 1.25 ↑0.09627 / 0.89150Depth accuracy at projected physical correspondence locations

Table 5: Inference intervention ablation on decoding geometric mechanisms using identical model weights

Inference Evaluation ConditionMean 3D Error (m) ↓Points w/ Error ≤ 1m (%) ↑Balanced Both-Correct Acc (%) ↑Depth AbsRel ↓RGB Recon PSNR ↑
DriveTok Nominal Model1.65067.5156.580.0962727.376
Intervention: Disable visibility mask2.48354.5826.890.2816719.407
Intervention: Zero-out Plücker ray embeddings2.04758.8449.680.1046026.385

Analysis of Cross-View Consistency

  1. Near-Field Precision and Far-Field Parallax Effects: Under real-world LiDAR projection, 67.51% of correspondence pairs maintain spatial deviations below 1.0 m, with depth AbsRel reaching 0.09627 in overlap zones. The difference between mean error (1.650 m) and RMS error (4.004 m) reflects sensor geometry: at distant ranges (40–50 m), baseline parallax and sparse point returns amplify small pixel-level depth errors upon 3D back-projection, forming an error tail.
  2. Cross-Camera Semantic Consensus: The cross-view semantic agreement reaches 88.38% (78.51% class-balanced), and the dual-view ground-truth accuracy reaches 77.80%. This indicates that the tokenizer assigns consistent semantic classifications to physical targets regardless of whether they appear in front or side camera frustums.
  3. Causal Mechanistic Validation: As shown in Table 5, disabling the visibility mask at inference time without retraining degrades cross-view 3D spatial error by +50.5% (rising to 2.483 m), while balanced accuracy drops from 56.58% to 26.89%. This confirms that cross-view consistency stems from physical ray queries and visibility constraints rather than shortcut fitting in 2D feature layers.
Reviewer_TM7WOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a 3D scene tokenizer that encodes multi-view images into scene tokens for various downstream autonomous driving tasks. While the overall framework is interesting, I have several major concerns regarding its novelty and experimental validation, as detailed below.

Critical strengths

1.Unifying multiple 3D perception tasks through a shared scene token representation is potentially useful.

2.The paper is generally well written and easy to follow.

Critical weaknesses
  1. Unclear novelty relative to existing BEV encoders.

The proposed “3D scene tokenizer” and “scene tokens” appear conceptually similar to existing BEV encoders and BEV features, as both map multi-view images to spatially structured scene representations. The paper should clarify the substantive differences in representation structure, training objectives, and information retained. Furthermore, controlled comparisons with representative BEV encoders on downstream tasks would help establish the benefits of the proposed representation. Such comparisons should use matched backbones, supervision, and training budgets where possible.

  1. Insufficient evaluation of reconstruction capability.

The experiments in the “Multi-View Reconstruction” section evaluate reconstruction quality only at the input camera views. These results provide limited evidence of the representation’s ability to maintain 3D consistency and generalize to unseen viewpoints. Novel-view synthesis experiments, using viewpoints excluded from the model inputs, would provide stronger validation of the claimed reconstruction capability.

  1. Mismatched supervision in the monocular depth estimation comparison.

The depth evaluation uses projected sparse LiDAR measurements as ground truth, while the depth prediction head is trained using pseudo-depth labels obtained by adjusting MoGe2 predictions with sparse LiDAR measurements. The proposed method therefore benefits indirectly from LiDAR supervision on the target dataset, whereas the baselines are evaluated in a zero-shot setting. This difference in supervision makes it difficult to attribute the reported performance gains to the proposed representation alone. The authors should clearly specify the data splits and the stages at which LiDAR measurements are used. They should also provide comparisons under matched supervision conditions, or an ablation that removes the LiDAR-based adjustment of the pseudo-depth labels.

  1. Incomplete comparison in occupancy prediction.

The paper claims to achieve performance comparable to state-of-the-art methods, but the occupancy evaluation omits relevant recent baselines, including VG3S (IROS 2026). The authors should include these comparisons and clarify any differences in temporal inputs, supervision, and evaluation protocols. Without a more comprehensive comparison under comparable settings, the claim of competitive performance is insufficiently substantiated.

Questions for authors

1.Could the authors clarify the technical differences between the proposed scene tokenizer and existing BEV encoders, and provide quantitative comparisons on representative downstream tasks, such as 3D object detection?

2.Could the authors compare the proposed method with depth estimation baselines under matched supervision and evaluation conditions?

3.Could the authors include comparisons with other stronger methods in the occupancy prediction experiments, while clearly stating any differences in experimental settings?

4.Could the authors add novel-view synthesis experiments to evaluate reconstruction quality at viewpoints excluded from the model inputs?

14 comments
AuthorsOpenReview ↗

Author Response - Part 1

W1 & Q1: Differences from Conventional BEV and Downstream 3D Understanding

1. Unifying Reconstruction and Understanding

DriveTok unifies reconstruction and understanding within one 3D scene representation. BEVFormer, OccNet, and HENet++ support multiple perception tasks, but their original objectives do not require reconstructing input images. DriveTok preserves appearance, metric geometry, and semantics in shared tokens for joint RGB, depth, semantic, and occupancy decoding.

This representation connects understanding and generation. Show-o integrates both within one model, SVG improves reconstruction while preserving perception performance and RAE supports reconstruction and diffusion modeling through pretrained semantic features. DriveTok extends this direction to multi-view 3D driving scenes.

The representation also supports joint future prediction. Built on DriveTok, Doe-2 (ECCV 2026) predicts future scene latents and decodes all four modalities under action conditioning. Section 4.2 of the Doe-2 paper states that “performing generation in an explicit unified 3D representation not only improves photorealistic temporal image synthesis but also preserves accurate metric-scale scene geometry”.

2. Implementation and Systematic Ablations

DriveTok combines joint objectives with geometrically constrained decoding. Deformable cross-attention aggregates six camera views into 1,024 scene tokens. Plücker ray embeddings and visibility constraints guide decoding. Table 1 evaluates these designs.

Table 1: Ablations on the nuScenes validation set

DimensionSetupPSNR ↑SSIM ↑Occupancy IoU (%) ↑Occupancy mIoU (%) ↑
Training ObjectivesRGB only (standard image tokenizer)28.840.77012.813.84
RGB + Depth28.520.76420.729.36
RGB + Depth + Semantics28.060.75624.0511.68
RGB + Depth + Semantics + Occupancy (Full)27.890.74733.3220.06
Geometric GuidanceWithout visibility attention mask29.980.7985.320.59
With visibility attention mask28.840.77012.813.84
Positional Encoding2D image positional encoding only24.690.63831.9819.39
3D Plücker ray encoding only23.670.60932.2119.47
2D positional encoding + 3D Plücker ray encoding25.240.66732.9619.53
Representation ArchitectureVGGT-based tokenizer25.450.70325.088.60
DriveTok scene tokenizer27.890.74733.3220.06
  • Joint supervision increases occupancy mIoU from 3.84% to 20.06% (+16.22 percentage points), while maintaining 27.89 dB reconstruction PSNR.
  • Visibility constraints preserve spatial information: under frozen linear probing after RGB-only pretraining, removing the mask reduces occupancy mIoU from 3.84% to 0.59%.
  • Image positions and camera rays are complementary: combining them improves both reconstruction and occupancy prediction.
  • The architecture improves both tasks: with identical heads and supervision, DriveTok exceeds the VGGT-based tokenizer by 2.44 dB PSNR and 11.46 percentage points in occupancy mIoU.

3. Downstream 3D Understanding on OpenScene

We connect DriveTok to Qwen3-VL-4B to evaluate spatial understanding on 11,389 non-overlapping OpenScene validation scenes:

  1. Zero-Shot: Off-the-shelf Qwen3-VL-4B with independent 2D views.
  2. Language-SFT: Qwen3-VL-4B fine-tuned on OpenScene QA and captioning with 2D inputs.
  3. DriveTok (Ours): Identical backbone and SFT protocol, replacing 2D image tokens with scene tokens.

Table 2: Downstream spatial understanding on OpenScene

Task CategoryTarget ObjectiveMetric (%)Zero-Shot Qwen3-VLLanguage-SFT 2D-VLMDriveTok (Ours)Absolute Gain vs. SFT
Spatial 3D VQANearest Agent OrientationAccuracy ↑ / Balanced Acc ↑32.45 / 28.1069.80 / 66.3587.21 / 85.40+17.41 / +19.05
Spatial 3D VQANearest Agent CategoryAccuracy ↑ / Balanced Acc ↑48.20 / 41.5081.65 / 78.4094.30 / 93.26+12.65 / +14.86
Spatial 3D VQARoadside Static ElementsAccuracy ↑ / Balanced Acc ↑36.80 / 25.4079.50 / 48.1593.60 / 59.95+14.10 / +11.80
Spatial 3D VQAVRU OrientationAccuracy ↑ / Balanced Acc ↑29.10 / 26.5065.20 / 61.7081.45 / 79.80+16.25 / +18.10
Dense Captioning3D Factuality AlignmentPrecision ↑ / Recall ↑ / F1 ↑45.30 / 21.80 / 29.4278.40 / 41.20 / 53.9891.50 / 54.27 / 68.13+13.10 / +13.07 / +14.15

DriveTok improves all evaluated tasks over 2D-input SFT, including +17.41 percentage points in agent-orientation accuracy and +14.15 points in captioning F1.

AuthorsOpenReview ↗

Author Response - Part 2

W2 & Q4: Task Scope Clarification, Sensor Dropout Robustness, and Physical Cross-View 3D Consistency

1. Task Scope and Design Objectives

The suggestion to evaluate representation quality via arbitrary Novel-View Synthesis (NVS) is an insightful perspective. Here, we clarify the academic scope and design goals of DriveTok:

  • Scope Differentiation: DriveTok focuses on learning a compact 3D scene tokenizer for autonomous driving decision-making, providing a unified representation for downstream planning, world model rollouts, and multi-task perception. It differs from generative rendering pipelines (such as NeRF, 3DGS, or dedicated NVS models) optimized for unconstrained trajectory novel-view synthesis.
  • Decoder Design: The decoder's view queries are conditioned on known vehicle camera extrinsics and calibration rays. Extending to arbitrary, uncalibrated viewpoints without dense multi-view pose supervision falls outside the current architectural design scope.

2. 3D Geometric Robustness Under Sensor Dropouts (Leave-One-Out Evaluation)

Although not trained for unconstrained view synthesis, to evaluate whether the model tolerates sensor failures and completes geometry using overlapping visual fields, we conducted a leave-one-out camera dropout test across all 6,019 nuScenes validation frames, evaluating geometry using 1,169,768 physical LiDAR projection point pairs:

Table 3: 3D physical geometry and depth retention under single-camera dropouts

Sensor Input ConditionMean 3D Error (m) ↓Points w/ Error ≤ 1m (%) ↑Balanced Both-Correct Acc (%) ↑Depth AbsRel ↓
Nominal Full Rig (All 6 cameras)1.65067.5156.580.09627
Front camera dropped (CAM_FRONT = 0)2.12459.3448.720.14210
Back camera dropped (CAM_BACK = 0)2.08660.1149.300.13850
Side camera dropped (Average across 4 sides)2.23557.8046.550.15120
Average across single-camera dropouts2.17058.5247.450.14620

When an entire camera stream is dropped, mean 3D spatial error rises moderately to 2.170 m, nearly 60% of points maintain a physical error below 1.0 m, and depth AbsRel remains around 0.14. Global 3D topology avoids divergence or collapse, showing that the model can leverage overlapping camera fields to support spatial continuity.

3. Quantitative Cross-View Geometric and Semantic Consistency via Physical Correspondences

To verify whether the representation maintains consistent 3D structure across cameras, we established a quantitative evaluation protocol: projecting the same physical LiDAR point onto two overlapping camera views across the nuScenes validation split (6,019 frames) yielded 1,169,768 physical correspondence pairs (1,164,033 with valid semantic labels). Predicted depths from both views were back-projected to world coordinates to measure Euclidean distance deviations and semantic agreement.

Table 4: Cross-view 3D physical consistency and decoder intervention ablation

Model Evaluation ConditionMean 3D Error (m) ↓Points w/ Error ≤ 1m (%) ↑Balanced Both-Correct Acc (%) ↑Depth AbsRel ↓RGB Recon PSNR ↑
DriveTok Nominal Model1.65067.5156.580.0962727.376
Intervention: Disable visibility mask2.48354.5826.890.2816719.407
Intervention: Zero-out ray embeddings2.04758.8449.680.1046026.385

Consistency Analysis:

  • Physical geometric agreement: 67.51% of overlapping observations have a predicted 3D deviation below 1.0 m, with an 88.38% semantic consistency rate (balanced both-correct accuracy is 56.58%). The mean 3D error of 1.650 m is influenced by parallax amplification at distant points (40–50 m), with higher accuracy in the near field.
  • Validation of architectural mechanisms: Disabling the visibility mask at test time increases cross-view 3D deviation by +50.5% (to 2.483 m) and reduces balanced accuracy to 26.89%. This indicates that visibility masking helps maintain consistent 3D structure across views.
AuthorsOpenReview ↗

Author Response - Part 3

W3 & Q2: Depth Supervision Clarification, Strict Data Splits, and Controlled Comparisons

Regarding the concern that depth evaluation might benefit from LiDAR pseudo-label calibration leading to mismatched comparisons with zero-shot baselines, we clarify the data usage protocol and report controlled experiments.

1. Data Split and LiDAR Usage Stages

  • Isolation of Splits: nuScenes follows the standard protocol: 28,130 training frames and 6,019 validation frames.
  • Supervision Isolation: LiDAR-based adjustment occurs solely during offline pseudo-label generation on the training split (using the training set's own LiDAR data to adjust MoGe-2 predictions). The LiDAR measurements of the 6,019 validation frames remain strictly read-only and are used exclusively for evaluation, without being accessed during training.

2. Controlled Comparison Without LiDAR Calibration

To isolate the effect of LiDAR adjustment on performance, we conducted a controlled ablation under the same training budget, comparing depth heads trained with unadjusted raw MoGe-2 pseudo-labels versus LiDAR-adjusted pseudo-labels, evaluated using the official read-only LiDAR projection protocol (3–50 m range, without test-time scaling).

Table 5: Impact of depth pseudo-label generation on metric depth estimation accuracy

Supervision Objective TypeIncludes Training LiDAR Scale CalibrationValidation AbsRel ↓Validation δ<1.25{\delta < 1.25} ↑
Unadjusted raw MoGe-2 pseudo-labelsNo0.170600.77925
LiDAR-adjusted MoGe-2 pseudo-labelsYes0.117390.86386
Paper final converged model (Full training)Yes0.081200.93240

Controlled Comparison Analysis: Even without training-stage LiDAR scale adjustments on depth pseudo-labels, DriveTok achieves an AbsRel of 0.17060 and a δ<1.25{\delta < 1.25} of 0.77925, showing solid metric depth prediction performance. For reference, this AbsRel is lower than R3D3 (0.25), SelfOcc (0.23), and OmniNWM (0.23) as reported in Table 3 of the paper. This indicates that depth targets without LiDAR adjustments still enable the model to learn meaningful metric depth representations.


W4 & Q3: Comparison with Recent Occupancy Methods and Clarification of Task Objectives

We apologize for incorrectly labeling these methods as temporal multi-frame methods. To prevent these errors from misleading reviewers, we have edited this response and corrected the erroneous entries. We thank the reviewer for bringing this issue to our attention.

The following table summarizes occupancy prediction results on the SurroundOcc benchmark.

Table 6: 3D semantic occupancy prediction results on SurroundOcc

MethodCore Task and RepresentationOccupancy IoU (%) ↑Occupancy mIoU (%) ↑
SurroundOccVoxel-based occupancy prediction31.4920.30
GaussianFormer-2Probabilistic Gaussian-based occupancy prediction30.5620.02
QuadricFormerSuperquadric-based occupancy prediction31.2220.12
VG3SGaussian occupancy prediction enhanced by geometric foundation models34.4121.52
StreamOccStreaming voxel-based occupancy prediction33.8023.40
DriveTok (Ours)Unified multimodal scene tokenizer33.3220.06

VG3S demonstrates the value of pretrained geometric priors for occupancy prediction. It extracts features using frozen foundation models and adapts them for Gaussian-based occupancy prediction through a hierarchical geometric feature adapter. The VG3S-DVGT reported in the table achieves 34.41 IoU and 21.52 mIoU. We have withdrawn our previous explanation linking its performance to temporal multi-frame inputs.

We claim that DriveTok only achieves comparable performance. Surpassing their occupancy prediction accuracy is not our research objective. DriveTok encodes multi-view images into 1,024 scene tokens that jointly support RGB reconstruction, depth estimation, semantic prediction, and 3D occupancy prediction. Under this joint-task setting, the model achieves 27.89 dB reconstruction PSNR, 0.080 depth AbsRel, and occupancy scores of 33.32 IoU and 20.06 mIoU.

VG3S and StreamOcc primarily target 3D semantic occupancy prediction, focusing on geometric and semantic prediction accuracy. DriveTok targets unified reconstruction and understanding, requiring the same compressed scene tokens to preserve appearance, geometry, and semantics and support joint decoding of the corresponding tasks.

Our goal is to unify reconstruction and understanding within a single 3D scene representation and use it as a visual interface for unified models and world models. Doe-2 (ECCV 2026), built on DriveTok, has demonstrated that this representation further supports joint prediction of future appearance, geometry, and semantics, while using the generated scene representations for downstream trajectory planning.

Reviewer_jG78OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DriveTok, a unified 3D scene tokenizer for multi-view autonomous driving. It leverages a pretrained vision foundation model and 3D deformable attention to encode multi-view images into scene tokens. A spatially aware multi-view decoder with visibility-guided attention supports RGB reconstruction, depth estimation, semantic prediction, and 3D occupancy prediction. Experiments on nuScenes demonstrate competitive performance across multiple tasks, with additional evaluations on occupancy forecasting and motion planning.

Critical strengths
  1. Meaningful problem setting. The paper addresses the challenge of learning compact representations that preserve both geometric and semantic information from multi-view driving scenes.
  2. Unified multi-task framework. DriveTok integrates reconstruction and scene understanding within a shared 3D representation, potentially benefiting downstream autonomous driving applications.
  3. Promising experimental results. The method achieves strong results in depth estimation, occupancy prediction, and downstream forecasting. The ablation studies also demonstrate the benefits of multi-task supervision and visibility-guided attention.
Critical weaknesses
  1. Limited methodological novelty. Although the paper addresses an interesting problem, my primary concern is its limited technical novelty. The framework mainly integrates existing techniques, including pretrained vision encoders, BEVFormer-style deformable attention, visibility-guided attention, and multi-task learning. The contributions beyond their integration remain unclear.
  2. Insufficient analysis of tokenization efficiency. The paper lacks a systematic analysis of token compression ratios and their impact on downstream performance. The efficiency advantages over existing multi-view representations also need further justification.
  3. Limited evaluation of 3D understanding. Despite claiming unified reconstruction and understanding, the paper mainly evaluates perception tasks, without exploring more challenging tasks such as 3D VQA and spatial reasoning.
  4. Incomplete discussion of related work. Relevant studies on geometric prediction and 3D scene understanding should be discussed and cited, including:
    • Unlocking Dense Metric Depth Estimation in VLMs
    • Omni-View: Unlocking How Generation Facilitates Understanding in Unified 3D Model based on Multiview Images
    • DepthEvidence: Unifying Metric Depth Prediction and Geometric Reasoning in Multimodal Language Models
Questions for authors
  1. What are the key methodological differences between DriveTok and existing BEV-based representations or geometry-aware driving tokenizers?

  2. Most experiments are conducted on nuScenes. How robust is DriveTok to different camera configurations, missing camera views, and varying input resolutions?

I am open to revising my rating based on the authors' rebuttal and feedback from other reviewers.

4 comments
AuthorsOpenReview ↗

Author Response -Part 1

We sincerely thank the reviewer for the thorough reading, professional evaluation, and constructive feedback. Below, we address each concern point by point with structured quantitative ablations, latency and runtime efficiency measurements, downstream multimodal reasoning evidence, and multi-camera robustness evaluations.


W1 & Q1: Methodological Differences Between DriveTok and BEV

1. Unifying Reconstruction and Understanding in a Single 3D Representation

DriveTok unifies reconstruction and understanding within a single 3D scene representation. BEVFormer, OccNet, and HENet++ have demonstrated the value of shared representations for detection, segmentation, occupancy prediction, and planning. Their original objectives, however, do not require recovering the input multi-view images. DriveTok requires the same scene tokens to preserve appearance and texture, metric geometry, and semantics, enabling joint decoding of RGB images, depth, semantic maps, and 3D occupancy.

This unification provides a shared representational foundation for understanding and generation. Show-o explores their integration within one model, while SVG and RAE demonstrate that semantically rich representations can support both reconstruction and generative modeling. SVG improves reconstruction while preserving classification, segmentation, and depth estimation performance, RAE combines pretrained visual encoders with reconstruction decoders and trains diffusion models in the resulting semantic latent space. DriveTok extends this direction to multi-view 3D driving scenes.

The representation also provides world models with a state space for jointly predicting appearance, geometry, and semantics. Built on DriveTok, Doe-2 (ECCV 2026) predicts future scene states in a unified 3D latent space and decodes them into multi-view RGB, depth, semantics, and occupancy. Section 4.2 of the Doe-2 paper explicitly states that “performing generation in an explicit unified 3D representation not only improves photorealistic temporal image synthesis but also preserves accurate metric-scale scene geometry”. This application demonstrates how a reconstructable, geometry-aware scene representation supports joint future scene modeling under action conditioning.

2. Implementation and Systematic Ablations

DriveTok combines joint reconstruction–understanding objectives with geometrically constrained multi-view decoding. Deformable cross-attention aggregates images from six cameras into 1,024 scene tokens, while Plücker ray embeddings and visibility constraints guide decoding. The central property we evaluate is whether these shared tokens simultaneously retain appearance details, metric geometry, and semantics. Table 1 examines the contributions of training objectives, visibility constraints, positional encoding, and representation architecture.

Table 1: Systematic ablations on the nuScenes validation set

Evaluation DimensionExperimental SetupPSNR ↑SSIM ↑Occupancy IoU (%) ↑Occupancy mIoU (%) ↑
Training ObjectivesRGB only (standard image tokenizer)28.840.77012.813.84
RGB + Depth28.520.76420.729.36
RGB + Depth + Semantics28.060.75624.0511.68
RGB + Depth + Semantics + Occupancy (Full)27.890.74733.3220.06
Geometric GuidanceWithout visibility attention mask29.980.7985.320.59
With visibility attention mask28.840.77012.813.84
Positional Encoding2D image positional encoding only24.690.63831.9819.39
3D Plücker ray encoding only23.670.60932.2119.47
2D positional encoding + 3D Plücker ray encoding25.240.66732.9619.53
Representation ArchitectureVGGT-based tokenizer25.450.70325.088.60
DriveTok scene tokenizer27.890.74733.3220.06
  • Joint supervision enriches geometry and semantics. Adding depth, semantic, and occupancy objectives increases occupancy mIoU from 3.84% to 20.06% (+16.22 percentage points), while maintaining 27.89 dB reconstruction PSNR.

  • Visibility constraints help preserve spatial information. Under frozen linear probing after RGB-only pretraining, removing the visibility mask reduces occupancy mIoU from 3.84% to 0.59%, supporting its role in spatial representation learning.

  • Image positions and camera rays provide complementary information. Combining 2D positional encoding with 3D Plücker ray encoding improves both reconstruction and occupancy prediction over either encoding alone.

  • The scene-token architecture improves reconstruction and understanding together. With identical task heads and supervision, DriveTok exceeds the VGGT-based tokenizer by 2.44 dB in PSNR and 11.46 percentage points in occupancy mIoU.

AuthorsOpenReview ↗

Author Response - Part 2

W2: Token Compression Ratio, Runtime Overhead, and Quality–Efficiency Trade-offs

A core advantage of DriveTok is that it decouples downstream sequence length from both camera count and input resolution. For a six-view camera rig with an input resolution of 256×704256 \times 704 (patch size 16), a standard view-by-view 2D tokenizer generates 4,224 tokens. In contrast, DriveTok compresses this input into a fixed budget of 1,024 scene tokens, achieving a 4.125×4.125\times sequence compression.

Downstream Multimodal LLM Overhead Comparison

When feeding compressed scene tokens into the downstream multimodal model (Qwen3-VL-4B), LLM decoding latency drops from 319.68 ms to 81.74 ms (a 74.4% latency reduction), and peak memory allocation decreases from 13,388.90 MB to 9,720.97 MB (a 27.4% memory saving).

Table 2: Downstream Qwen3-VL-4B inference overhead under different input representations

Input Representation TypeNumber of TokensLLM Latency (ms) ↓LLM Peak Memory (MB) ↓
Direct 2D Patch Representation4,224319.6813,388.90
BEV-VAE1,02481.749,720.97
Triplane1,536122.1010,301.02
DriveTok (Ours)1,02481.749,720.97

System-Level Benchmarking Across Token Budgets

To evaluate the trade-off among sequence length, execution latency, and representation fidelity, we benchmarked multiple token budgets on a standardized hardware platform (single NVIDIA A800-80GB GPU, batch size 1) with an identical early-stage training budget (3,520 parameter updates).

Table 3: Unified latency, memory footprint, and early representation quality across different token budgets

Token BudgetTokenization Latency (ms) ↓Full-Task Forward Latency (ms) ↓Peak Memory (MiB) ↓PSNR ↑SSIM ↑Occupancy IoU (%) ↑Occupancy mIoU (%) ↑LLM First-Token Latency (ms) ↓
25642.99399.086,493.3821.6580.591027.9513.2032.57
1,02443.00410.426,481.6022.0710.601428.4115.4368.15
4,09643.18445.676,529.1622.2580.607328.7515.87242.98

Note: Latency and memory measurements were obtained using BF16 precision with FP32 BEV attention, keeping all task heads active. Quality metrics were evaluated on the epoch-4 validation set.

Efficiency Trade-off Analysis

  1. Constant front-end latency: Across all configurations, tokenization latency remains stable at approximately 43 ms with ~6,500 MiB memory usage. This confirms that front-end feature extraction and BEV deformable cross-attention dominate the primary compute budget and are largely decoupled from the output token count.
  2. 1,024 tokens balance efficiency and quality: Scaling from 256 to 1,024 tokens provides a substantial boost in occupancy mIoU (from 13.20% to 15.43%, a net improvement of +2.23 percentage points), while downstream LLM first-token latency remains low at 68.15 ms. In contrast, scaling further to 4,096 tokens yields marginal perception gains (mIoU increases by only +0.44 percentage points) while downstream LLM response latency spikes to 242.98 ms (3.56×3.56\times degradation). These findings confirm that 1,024 tokens effectively balance representation quality and end-to-end system throughput.
AuthorsOpenReview ↗

Author Response - Part 3

W3: Quantitative Validation of 3D Understanding and Complex Spatial Reasoning

At the foundational perception level, DriveTok demonstrates strong spatial preservation on the nuScenes validation set: metric depth achieves an AbsRel of 0.08 (δ<1.25\delta < 1.25 of 0.93), 3D semantic occupancy reaches an IoU of 33.32% (mIoU 20.06%), and 2D semantic segmentation attains an mIoU of 68.9% (markedly outperforming BEVFormer's 56.2% and TPVFormer's 59.3%).

Downstream Autoregressive Multimodal Evaluation on OpenScene

To directly address the reviewer's concern regarding higher-level 3D spatial reasoning (such as 3D VQA and dense scene captioning), we trained a specialized DriveTok representation model on the OpenScene dataset and interfaced its scene tokens with an autoregressive multimodal backbone (Qwen3-VL-4B). Evaluations were conducted on a challenging validation split comprising 11,389 non-overlapping real-world scenes.

Baseline Definitions and Controlled Experimental Setup

To guarantee fair comparisons, three distinct configurations were evaluated under identical multi-view inputs and standardized evaluation protocols:

  1. Zero-Shot Baseline: Evaluates the off-the-shelf Qwen3-VL-4B model using independent 2D view inputs with standard system prompting, representing the native 3D spatial reasoning capabilities of general-purpose foundation models.
  2. Language-SFT Baseline: Uses the same Qwen3-VL-4B backbone with direct 2D image inputs, supervised via full instruction fine-tuning (SFT) on the OpenScene 3D question-answering and captioning splits. This represents standard vision-language approaches lacking explicit 3D scene tokenization.
  3. DriveTok Representation (Ours): Keeps the downstream autoregressive LLM backbone (Qwen3-VL-4B), SFT training data, and hyperparameters strictly identical. The only controlled variable is replacing the raw 2D image tokens with DriveTok's unified 3D scene tokens.

Table 4: Comparative evaluation of DriveTok and baselines on OpenScene 3D spatial reasoning tasks

Task CategoryTarget Objective / DimensionMetric (%)Zero-Shot Qwen3-VLLanguage-SFT 2D-VLMDriveTok 3D Tokens (Ours)Relative Gain vs. SFT
Spatial 3D VQANearest Agent OrientationAccuracy ↑ / Balanced Acc ↑32.45 / 28.1069.80 / 66.3587.21 / 85.40+17.41 / +19.05
Spatial 3D VQANearest Agent CategoryAccuracy ↑ / Balanced Acc ↑48.20 / 41.5081.65 / 78.4094.30 / 93.26+12.65 / +14.86
Spatial 3D VQARoadside Static ElementsAccuracy ↑ / Balanced Acc ↑36.80 / 25.4079.50 / 48.1593.60 / 59.95+14.10 / +11.80
Spatial 3D VQAVRU OrientationAccuracy ↑ / Balanced Acc ↑29.10 / 26.5065.20 / 61.7081.45 / 79.80+16.25 / +18.10
Dense Captioning3D Factuality AlignmentPrecision ↑ / Recall ↑ / F1 ↑45.30 / 21.80 / 29.4278.40 / 41.20 / 53.9891.50 / 54.27 / 68.13+13.10 / +13.07 / +14.15

Key Empirical Findings

  • Consistent gains over 2D instruction fine-tuning: The zero-shot baseline struggles to infer 3D orientations without explicit geometric structure. While language-based SFT improves conversational alignment, it reaches performance plateaus on tasks requiring rigorous metric orientation and spatial layout reasoning.
  • Explicit 3D tokens enable complex spatial reasoning: Driven by DriveTok scene tokens, the model achieves 87.21% accuracy in nearest agent orientation prediction (+17.41% over Language-SFT) and an F1-score of 68.13 in dense captioning (+14.15 improvement). This shows that compact 3D geometric tokens effectively resolve the spatial ambiguity common in conventional 2D-VLMs.
  • Closed-loop motion planning validation: In the NAVSIM benchmark, an end-to-end planner driven by DriveTok tokens achieves 94.0 PDMS and 90.8 EPDMS, outperforming the baseline planner by +6.5 PDMS and confirming the representation's utility in downstream decision-making.
AuthorsOpenReview ↗

Author Response - Part 4

W4: Systematic Discussion and Categorization of Related Literature

Table 5 details the architectural distinctions between DriveTok and relevant studies (DepthVLM, Omni-View, and DepthEvidence).

Table 5: Systematic comparison with recent geometric perception and 3D understanding works

ModelCore Problem FormulationSensor Input ScopeLatent RepresentationDownstream Interface
DepthVLMMonocular dense depth estimation via VLMsMonocular single view2D image patch tokensDense 2D depth maps
Omni-ViewJoint understanding, NVS, and reconstructionMulti-view camera arrayHybrid view queriesNovel view rendering / 3D perception
DepthEvidenceObject-centric metric bounding boxes for reasoningMonocular / stereo pairsSparse object bounding tokensLanguage-guided geometric QA
DriveTok (Ours)Unified 3D scene tokenization for autonomous drivingSurround multi-camera rigFixed 1,024 compact 3D metric scene tokensDecoupled execution: RGB, depth, occupancy, QA, planning

Core Design Distinctions

  • Surround metric space aggregation vs. monocular processing: DepthVLM and DepthEvidence primarily address single-view or localized observations. DriveTok explicitly leverages camera intrinsics and extrinsics to lift multi-view observations into a shared 3D metric space, resolving cross-camera redundancy and metric scale ambiguities.
  • Unified decoupled state vs. task-specific decoders: Omni-View tightly couples view queries with rendering targets. DriveTok distills sensory inputs into a resolution-agnostic, unified 3D latent state that simultaneously supports dense occupancy forecasting, surface reconstruction, and closed-loop planning, while allowing expensive rendering decoders to be bypassed during on-vehicle deployment.

Q2: Robustness Across Camera Configurations, View Dropouts, and Resolutions

1. Robustness to Camera Rig Configurations

DriveTok is grounded in metric vehicle coordinates, projecting ray queries using calibrated camera intrinsics KiK_i and extrinsics [Ri∣ti][R_i \mid t_i]. In commercial production, sensor rig geometries remain rigid per platform, making platform-specific models standard practice. Importantly, as detailed in Figure 11 and Table 11 of our submission, DriveTok trained on nuScenes generalizes zero-shot to PandaSet (which features a different sensor layout) without any fine-tuning, achieving an AbsRel of 0.289 compared to DVGT's 0.405.

2. Geometric Robustness Under Camera Failure (Leave-One-Out Evaluation)

To quantify robustness under camera occlusions or hardware dropouts, we conducted a leave-one-out evaluation across all 6,019 nuScenes validation frames, measuring 3D error using 1,169,768 LiDAR-projected correspondence pairs:

Table 6: 3D physical geometry and depth estimation under single-camera failure

Input ConditionMean 3D Error (m) ↓Points w/ Error ≤ 1m (%) ↑Balanced Both-Correct Acc (%) ↑Depth AbsRel ↓
Nominal Full Rig (All 6 cameras)1.65067.5156.580.09627
Front camera dropped (CAM_FRONT = 0)2.12459.3448.720.14210
Back camera dropped (CAM_BACK = 0)2.08660.1149.300.13850
Average across single-camera dropouts2.17058.5247.450.14620

Analysis: When an entire camera stream is dropped, mean 3D spatial error increases moderately from 1.650 m to 2.170 m. Nearly 60% of points maintain a physical error below 1.0 m, and depth AbsRel remains stable around 0.14. The 3D scene tokens prevent catastrophic structural failure by leveraging redundant geometric constraints from adjacent views.

3. Cross-View Physical Consistency and Decoding Mechanism Ablation

We report metric 3D error and semantic agreement across corresponding physical observation points:

Table 7: Cross-view 3D consistency and decoder intervention ablation

Model ConfigurationMean 3D Error (m) ↓Points w/ Error ≤ 1m (%) ↑Balanced Both-Correct Acc (%) ↑Depth AbsRel ↓RGB Recon PSNR ↑
DriveTok Nominal Model1.65067.5156.580.0962727.376
Disable visibility mask2.48354.5826.890.2816719.407
Zero-out ray embeddings2.04758.8449.680.1046026.385

Mechanistic Validation: The nominal DriveTok model achieves an 88.38% semantic consistency rate across overlapping camera views, with 67.51% of points exhibiting 3D deviations under 1.0 m. Disabling the visibility mask at test time increases 3D spatial error by +50.5% (to 2.483 m) and drops balanced accuracy to 26.89%. This provides direct empirical evidence that visibility masking and Plücker ray embeddings are essential for maintaining cross-view geometric coherence and resolving view conflicts.

Comments on the paper

2 comments

Thank you for your kindly feedback and for taking the time to read our response. We’re glad our additional experiments and clarifications addressed your concerns.

AuthorsOpenReview ↗

Summary of the Author–Reviewer Discussion

Dear Program Chairs, Senior Area Chairs, Area Chairs, and Reviewers,

Thank you for your careful review and constructive feedback. We appreciate the reviewers’ recognition of two strengths of DriveTok:

  • A unified representation for multi-view driving scenes. DriveTok encodes six camera views into 1,024 spatially anchored scene tokens. The same tokens support image reconstruction, metric depth estimation, semantic segmentation, and 3D occupancy prediction. On nuScenes, DriveTok achieves 33.32% IoU and 20.06% mIoU for current-frame occupancy.
  • A shared representation for world modeling and planning. DriveTok-World predicts future scene tokens, jointly decodes them into multi-view RGB, depth, semantics, and 3D occupancy, and uses the predicted representation for trajectory planning.

During the discussion, we added the following experiments and clarifications:

  • Task scope. We clarified that multi-view reconstruction recovers the input camera images and that the scene tokens are organized in 3D space. (jG78, TM7W, AGe9)
  • Representation comparisons and ablations. We compared DriveTok with VGGT and BEV representations and examined the effects of multi-task supervision, visibility constraints, and camera-ray embeddings. (jG78, TM7W, AGe9)
  • Cross-view consistency and robustness. Across 6,019 nuScenes validation frames and approximately 1.17 million LiDAR point correspondences, predictions in overlapping camera views had a mean 3D discrepancy of 1.65 m and 88.38% semantic agreement. We also evaluated single-camera dropout and occupancy decoding after aligning tokens using ego motion, and discussed zero-shot depth transfer to PandaSet’s six-camera setup. (jG78, TM7W, AGe9)
  • Depth supervision and occupancy evaluation. We clarified how LiDAR is used in training and validation. Under the same training budget, calibrating MoGe-2 pseudo-labels with LiDAR reduced depth AbsRel from 0.17060 to 0.11739. We also corrected temporal-input labels for several baselines and clarified the evaluation differences among SurroundOcc, OpenOcc, and Occ3D. (TM7W, AGe9)
  • Token budget and efficiency. Under the same early training budget, increasing the token count from 256 to 1,024 improved occupancy mIoU by 2.23 percentage points. Increasing it to 4,096 added 0.44 points. In the Qwen decoder test, time to first token was 68.15 ms with 1,024 tokens and 242.98 ms with 4,096. (jG78)
  • Forecasting and spatial reasoning. We detailed DriveTok-World’s spatial and temporal attention design. With the same forecasting architecture and a budget of 1,024 tokens, DriveTok improved mean future occupancy mIoU and IoU over the BEV baseline by 5.14 and 18.43 percentage points, respectively. We also evaluated spatial question answering and scene description on OpenScene using DriveTok with Qwen3-VL-4B. (jG78, TM7W, AGe9)

Thank you again for your time. Your feedback helped us clarify what information DriveTok’s scene tokens preserve and how that information supports downstream tasks.

Best regards,
The authors of Submission 23013

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.