acceptodds
Under review as a conference paper at ICLR 2027

DVGT-2: Vision-Geometry-Action Model for Autonomous Driving

Abstract

Vision-Language-Action (VLA) and World-Action (WA) models offer promising routes to end-to-end autonomous driving. In this paper, we introduce an alternative Vision-Geometry-Action (VGA) paradigm and demonstrate that dense geometric supervision can support competitive driving planning. As vehicles operate in a 3D world, dense geometry provides rich spatial cues for decision-making. However, the computational overhead of multi-frame geometry reconstruction limits its use in online driving. To address this, we introduce a streaming Driving Visual Geometry Transformer (DVGT-2), which jointly predicts local geometry and future trajectories, aligning geometric reconstruction with ego-centric planning. We employ temporal causal attention and cache historical features to avoid repetitive computation. We further propose a sliding-window streaming strategy and use relative temporal encoding to support online inference with bounded per-frame computation and memory. Experiments demonstrate superior local geometric accuracy across five driving datasets and substantially improved online reconstruction efficiency. DVGT-2 achieves strong planning performance on NAVSIM v1/v2 and remains competitive on nuScenes, while offering low inference latency compared with representative VLA and WA models.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Autonomous Driving, Visual Geometry Model, Vision Language Action Model

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_SprnOpenReview ↗
Rating
3
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates whether dense geometric supervision can provide useful scene representations for camera-based autonomous driving. It introduces DVGT-2, a DINOv3-based multi-view model that jointly predicts ego-centric local pointmaps, relative ego-poses, and future ego-trajectories through shared features. Its streaming implementation combines factorized spatial attention, temporal causal attention, a fixed-size per-layer feature cache, and relative temporal encoding. Predicting local geometry and relative motion avoids anchoring reconstruction to the first frame, allowing older cached features to be discarded. Training uses MoGe-2-derived geometric supervision, followed by joint geometry–planning training and benchmark-specific fine-tuning. Validation includes geometry reconstruction on five driving datasets, planning on NAVSIM v1/v2 and nuScenes, component and cache-size ablations, and long-sequence accuracy and efficiency measurements. The reported results show strong planning performance and improved local ray-depth accuracy, with bounded per-frame computational and memory costs.

Critical strengths

S1. A coherent streaming design with meaningful long-sequence validation. The combination of ego-centric geometry, relative motion prediction, bounded feature caching, and relative temporal encoding addresses a concrete limitation of online multi-frame geometry models: increasing computation and memory as the observation history grows. Importantly, the evidence is not limited to an asymptotic argument or short clips. Figure 5 reports approximately constant per-frame costs over more than 1,200 frames.

S2. Strong task-level results make geometry-supervised planning a credible alternative. DVGT-2 reports 92.5 PDMS on NAVSIM v1 and 90.0 EPDMS on NAVSIM v2, the highest aggregate scores among the methods listed in Tables 3 and 4. On nuScenes, it reports an average displacement error of 0.78 m and an average collision rate of 0.19%. These results are substantial evidence that the complete geometry-supervised system can perform competitive driving planning, rather than merely improve a reconstruction objective. This empirical effectiveness is the strongest argument for the paper’s significance, although it does not by itself establish which component causes the gains.

Critical weaknesses

W1. The contribution of dense geometric supervision is not sufficiently isolated in the final streaming planner. Table 6 provides important supporting evidence: adding geometric supervision increases PDMS from 71.2 to 91.5 in the non-streaming configuration, and the authors explicitly state that the variants use the same training data and budget. However, there is no corresponding geometry-supervision ablation for the complete streaming model in row f. Consequently, the paper does not directly establish the contribution of geometric supervision to the final model’s headline benchmark results.

W2. There is a lack of experimental evidence demonstrating the relationship between window size W and planning performance. Table A.5 motivates the choice of W=3 using reconstruction accuracy and resource consumption, without reporting planning performance across window sizes. Thus, the selected history budget is justified for geometry reconstruction but not directly for the paper’s main driving task. The bounded-memory property is convincing; the planning-specific benefit and trade-off of the chosen streaming configuration are less fully established.

Other minor issues: W3. In Table 5, under the “Collision Rate” metric, the “Avg” value for DVGT-2 is both underlined and bolded—is this a mistake?

Questions for authors

Q1 — Clarification and targeted validation of the geometric-supervision claim. How is the no-geometry variant in Table 6 trained during the stage corresponding to the 160K-iteration geometry pretraining, and does it retain pose supervision? Can the authors provide a comparison between the complete streaming model and a counterpart without dense geometric supervision, keeping the backbone, planning head, training data, and optimization budget matched, preferably on a standard NAVSIM split? This would directly affect my assessment of whether the final planning gains can be attributed to dense geometric supervision.

Q2 — Targeted validation of the streaming claim. How does the proposed cached model compare against a bounded-history recomputation baseline using the same recent-frame budget, local-coordinate targets, and relative temporal encoding? Also, how do planning scores and comfort change with cache-window size? These comparisons would clarify whether the selected streaming configuration provides a favorable planning accuracy–efficiency trade-off beyond avoiding full-cache resets. Table A.5 currently reports only geometry accuracy and resource measurements.

Q3 — Clarification of benchmark evaluation and cached state. For the NAVSIM results, particularly Table 4, please specify the exact devkit version and split/scenario configuration, together with how cached state is initialized, carried forward, and reset. Is the cache reconstructed independently for each evaluated scenario, or retained across consecutive scenarios from the same log? How is the admissible observation history kept comparable with the baselines? These details materially affect the interpretation and reproducibility of the reported scores for a stateful streaming model.

Q4 — Clarification of geometric evaluation targets. What are the sources of the reference point clouds and ray depths used for geometry evaluation in Section C? Are these sensor-based measurements or model-generated labels, and are they independent of the MoGe-2-derived training supervision?

3 comments
AuthorsOpenReview ↗

Author Response - Part 1/2

We thank the reviewer for constructive suggestions to our paper. We provide our response below.

R1. Geometric supervision in the complete streaming planner (W1, Q1).

In Table 6, the planning-only variant (row a) received neither pointmap nor pose supervision and therefore skipped the 160K-step geometry pretraining stage. Row b completed that stage, so the training budgets matched only for the two subsequent stages.

To isolate the effect of geometric supervision in the complete streaming DVGT-2, we trained two variants with the same backbone, planning head, initial weights, training data, and optimization schedule. Both skipped geometry pretraining and underwent 80K steps of streaming training followed by 40K steps of NAVSIM fine-tuning. Across both stages, one variant used only the planning loss, while the other also used pointmap and relative ego-pose losses. On NAVSIM v1 navtest, the full streaming DVGT-2 achieved 90.4 PDMS with additional geometric supervision, compared with 72.6 PDMS with planning supervision alone.

Training lossesNAVSIM v1 navtest PDMS ↑
Planning w/o Geometry72.6
Planning w/ Geometry90.4

R2. Cached inference versus recomputation at a fixed history window (Q2).

We compared two inference modes of the same DVGT-2 checkpoint on NAVSIM v1 navtest, keeping the history window (W=3), local-coordinate targets, and relative temporal encoding fixed. At each step, the baseline recomputed features for the current frame and the three preceding frames, while the cached model reused the preceding frames’ features to process the current frame. At the same history window, the full streaming DVGT-2 achieves better efficiency without sacrificing accuracy.

ModelInference modeAbs Rel ↓PDMS ↑Latency (ms) ↓Peak GPU memory (GB) ↓
DVGT-2Recompute window0.04492.564918.9
DVGT-2Reuse cached features0.04492.526816.5

R3. Effect of cache window size on planning (W2, Q2).

We evaluated the same DVGT-2 checkpoint with different window sizes (W) on the 200 OpenScene test sequences used in Table 6. We used these long sequences because the standard NAVSIM v1 navtest scenarios provide only the current frame and three preceding frames, limiting the history only available to W=3. Windows larger than W=3 increase latency without consistently improving PDMS, and Comfort remains at 1.00 across all window sizes.

MetricW=1W=3W=5W=7W=9
PDMS ↑91.6992.1792.0092.1991.90
Comfort ↑1.001.001.001.001.00
Latency (ms) ↓263278286294303
AuthorsOpenReview ↗

Author Response - Part 2/2

R4. NAVSIM evaluation protocol and cache state (Q3).

We used devkits based on NAVSIM releases v1.1 for Table 3 and v2.2 for Table 4. For each version, we evaluated all scenarios in the official navtest split. We evaluated each scenario independently and built its cache from scratch, with no cache retained across consecutive scenarios from the same driving log. Following the standard NAVSIM evaluation protocol, DVGT-2 used only the current frame and the three preceding frames (W=3), matching the observation history available to the baselines.

R5. Geometric evaluation targets (Q4).

In Section C, following DVGT, we use the same MoGe-2 pseudo-labeling pipeline to generate the geometric targets for training and evaluation. We use pseudo-labels because LiDAR measurements in driving datasets are too sparse to provide dense geometric targets. As an independent check, we also compare DVGT-2’s predictions directly with raw LiDAR measurements on OpenScene and nuScenes. Although trained on pseudo-labels, DVGT-2 reconstructs scene geometry more accurately than VGGT on both datasets when evaluated directly against raw LiDAR measurements.

MethodOpenScene Acc ↓OpenScene Abs Rel ↓nuScenes Acc ↓nuScenes Abs Rel ↓
VGGT1.170.211.470.26
DVGT-20.720.110.840.11
MoGe-2 pseudo-labels0.650.110.700.10

R6. Typo (W3). The underline was a typo. The lowest value (0.19%) should be in bold only. We have fixed it in the revised paper.

Dear Reviewer Sprn,

Thank you for your thoughtful review and constructive suggestions, which have significantly improved the quality and clarity of our paper. We sincerely hope that our rebuttal has fully addressed all of your concerns. As the discussion period is ending soon, if you need any further clarification or additional information, please do not hesitate to contact us. We truly appreciate your time and effort devoted to reviewing our work.

Best regards, Authors

Reviewer_U7DhOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces DVGT-2, a Vision-Geometry-Action model that jointly predicts dense local pointmaps, relative ego-poses, and future driving trajectories from multi-view images. Its main architectural contribution combines ego-centric geometry prediction, temporal causal attention, a fixed-size feature cache, and relative temporal encoding to support streaming inference with bounded per-frame computation and memory. The model is pretrained on five driving datasets and subsequently fine-tuned for planning. Validation includes geometry reconstruction across five datasets, NAVSIM v1/v2 planning, nuScenes open-loop planning, component ablations, and long-sequence efficiency measurements. The reported results demonstrate strong local geometry accuracy and competitive planning performance.

Critical strengths
  1. A coherent solution to the streaming geometry–planning problem. Predicting local pointmaps and relative motion makes fixed-window caching compatible with continuous inference. Figure 5 and Tables A.5–A.6 support bounded memory and computation, together with sustained local prediction quality over long sequences. This is consequential because it addresses a practical limitation of applying full-sequence geometry models to online driving.
  2. Substantial empirical evidence that geometry-supervised representations can support planning. DVGT-2 reports PDMS 92.5 on NAVSIM v1 and EPDMS 90.0 on NAVSIM v2, alongside improved local depth metrics across all five geometry datasets. Table 6 also shows a large planning improvement when geometric supervision is introduced. These results make the proposed approach practically interesting beyond an efficiency-oriented reconstruction model.
Critical weaknesses
  1. The central geometry-to-planning claim is not fully isolated in the final architecture. Table 6 compares geometric supervision using the non-streaming variants, where PDMS increases from 71.2 to 91.5, but does not include the final streaming architecture trained without geometric supervision. Consequently, it remains unclear how much geometry contributes under the configuration used for the principal results. Comparisons against other paradigms also differ in pretraining, architecture, and supervision. This limits the strength of the claim that dense geometry itself is the decisive source of the planning gains.
  2. The efficiency evidence establishes bounded scaling more convincingly than superiority over equally bounded alternatives. Full-sequence baselines repeatedly process the observed history, while full-history streaming variants eventually require cache resets. These comparisons demonstrate the benefit of restricting history, but do not isolate DVGT-2 against a strong baseline using the same window, backbone, and input budget. Table 7 also uses heterogeneous view counts and resolutions. This matters because the claimed architectural advance needs to be distinguished from the general advantage of processing less history.
Questions for authors
  1. Clarification and targeted validation. What planning performance does the complete streaming architecture achieve when trained without geometric supervision, using the same initialization, training data, optimization budget, and planning head? This comparison would directly affect my assessment of the central VGA contribution.
  2. Clarification of the streaming comparison. Were DVGT or another geometry baseline evaluated using the same fixed historical window and comparable temporal encoding? If available, please report the resulting accuracy–latency trade-off. This would clarify which improvements arise from the proposed formulation rather than history truncation alone.
  3. Clarification of supervision and evaluation independence. Are geometric evaluation targets derived from independent sensor measurements or from the MoGe-2 pseudo-labeling pipeline? Also, how is log-level separation ensured between OpenScene pretraining data and NAVSIM evaluation data? These details are important for interpreting reconstruction quality and downstream generalization.
3 comments
AuthorsOpenReview ↗

Author Response - Part 1/2

We thank the reviewer for constructive suggestions to our paper. We provide our response below.

R1. Geometric supervision in the full streaming DVGT-2 (W1, Q1).

We trained two variants of the full streaming DVGT-2 architecture, one with and the other without the geometry supervision.

For a fair comparison, both variants skipped the initial 160K-step geometry pretraining stage and started from identical weights. They used the same planning head, training data, and optimization schedule. Each variant retained its respective losses through 80K steps of streaming training followed by 40K steps of NAVSIM fine-tuning.

On NAVSIM v1 navtest, the full streaming DVGT-2 achieved 90.4 PDMS with additional geometric supervision, compared with 72.6 PDMS with planning supervision alone.

Training lossesNAVSIM v1 navtest PDMS ↑
Streaming w/o Geometry72.6
Streaming w/ Geometry90.4

R2. Geometry accuracy and latency at a fixed history window (W2, Q2).

We evaluated DVGT and DVGT-2 on the same eight-view sequences from the NAVSIM v1 navtest, using a fixed history of three frames (W=3) and each model’s native temporal encoding. To isolate the benefit of our streaming design, we evaluated the same DVGT-2 weights in two modes: recomputing the current and preceding three frames at every step, or reusing cached features from the preceding frames. At the same history window, the full streaming DVGT-2 achieves better accuracy and efficiency.

ModelInference modeAbs Rel ↓PDMS ↑Latency (ms) ↓Peak GPU memory (GB) ↓
DVGTRecompute window0.047-63918.8
DVGT-2Recompute window0.04492.564918.9
DVGT-2Reuse cached features0.04492.526816.5
AuthorsOpenReview ↗

Author Response - Part 2/2

R3. Geometric evaluation targets (Q3).

Following DVGT, we use the same MoGe-2 pseudo-labeling pipeline to generate the geometric targets for training and evaluation. We use pseudo-labels because LiDAR measurements in driving datasets are too sparse to provide dense geometric targets. As an independent check, we also compare DVGT-2’s predictions directly with raw LiDAR measurements on OpenScene and nuScenes. Although trained on pseudo-labels, DVGT-2 reconstructs scene geometry more accurately than VGGT on both datasets when evaluated directly against raw LiDAR measurements.

MethodOpenScene Acc ↓OpenScene Abs Rel ↓nuScenes Acc ↓nuScenes Abs Rel ↓
VGGT1.170.211.470.26
DVGT-20.720.110.840.11
MoGe-2 pseudo-labels0.650.110.700.10

R4. Log-level separation (Q3). All OpenScene data used for pretraining came from the OpenScene trainval split. We evaluated on NAVSIM navtest, which is drawn from the OpenScene test split and shares no driving logs with trainval.

Dear Reviewer U7Dh,

Thank you for your thoughtful review and constructive suggestions, which have significantly improved the quality and clarity of our paper. We sincerely hope that our rebuttal has fully addressed all of your concerns. As the discussion period is ending soon, if you need any further clarification or additional information, please do not hesitate to contact us. We truly appreciate your time and effort devoted to reviewing our work.

Best regards, Authors

Reviewer_atTcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DVGT-2, a streaming Vision-Geometry-Action framework for autonomous driving that jointly performs dense geometry reconstruction, relative ego-pose estimation, and trajectory planning. The method introduces ego-centric local geometry prediction, fixed-window feature caching, and relative temporal encoding to support efficient online inference. Experiments demonstrate strong geometric reconstruction performance across five driving datasets and competitive planning results on NAVSIM and nuScenes, while maintaining approximately constant computational and memory costs during long-sequence inference.

Critical strengths
  1. Strong planning performance supported by extensive evaluations. DVGT-2 achieves 92.5 PDMS on NAVSIM v1 and 90.0 EPDMS on NAVSIM v2, outperforming the reported competing methods. It also achieves competitive planning accuracy on nuScenes. These results provide substantial evidence that geometry-supervised representations can support effective autonomous driving planning.

  2. Effective and computationally efficient streaming architecture. The proposed fixed-window streaming strategy enables online geometry reconstruction and trajectory prediction with bounded memory and computational costs. The long-sequence experiments demonstrate relatively stable prediction quality and inference latency, addressing an important practical limitation of existing geometry reconstruction models.

Critical weaknesses
  1. Limited methodological novelty and insufficient isolation of the contribution of dense geometry. The proposed framework largely integrates existing components, including pretrained geometry transformers, sliding-window caching, relative temporal encoding, and diffusion-based planning heads. Although the ablation studies demonstrate a substantial benefit from geometry supervision, they do not sufficiently establish whether dense pointmap reconstruction offers advantages over simpler geometric supervision, such as depth or ego-pose prediction. This makes it difficult to determine which aspects of the proposed VGA framework are essential to the reported improvements.

  2. Concerns regarding experimental fairness and evaluation protocols. DVGT-2 uses a large pretrained backbone, extensive multi-dataset pretraining, and eight-camera temporal inputs, whereas the compared planning methods operate under heterogeneous input and training settings. Consequently, the reported performance gains cannot be attributed solely to the proposed framework. Furthermore, the key planning ablations are conducted on a nonstandard OpenScene evaluation rather than the official NAVSIM navtest split, and the streaming ablations involve baselines affected by cache resets. Stronger controlled comparisons are needed to substantiate the central claims.

Questions for authors
  1. Can the authors provide controlled comparisons between dense pointmap supervision and simpler geometric objectives, such as depth-only or ego-pose-only supervision, using the same architecture and training budget? This would clarify whether dense geometry reconstruction is essential to the proposed VGA paradigm.

  2. How much of the improvement on NAVSIM can be attributed to the proposed architecture rather than the additional camera views, temporal context, large-scale pretraining, or model capacity? Could the authors provide comparisons under more closely matched input configurations and training budgets?

  3. Could the authors clarify how PDMS is computed on the OpenScene long-sequence evaluation used in Table 6? Additionally, can the effects of local-coordinate reconstruction, fixed-window caching, and relative temporal encoding be isolated using matched streaming configurations without confounding cache-reset behavior?

  4. Since NAVSIM is derived from OpenScene, can the authors explicitly confirm recording-level separation between OpenScene pretraining and NAVSIM evaluation? Furthermore, have the authors evaluated DVGT-2 under reactive planning conditions, such as NAVSIM navhard, to assess whether its streaming geometric representations remain effective when the environment evolves in response to the agent?

3 comments
AuthorsOpenReview ↗

Author Response - Part 1/2

We thank the reviewer for constructive suggestions to our paper. We provide our response below.

R1. Novelty and the role of dense geometry (W1, Q1).

Our main contribution is the Vision-Geometry-Action (VGA) framework. Since driving requires planning in 3D, VGA uses dense 3D reconstruction as its primary perception task to learn spatial-aware visual representations for planning. We supervise dense 3D reconstruction with pseudo-labels generated automatically from sensor data, avoiding additional manual annotation. Together with VGA’s support for different camera configurations, this makes it easier to scale training across driving datasets.

To evaluate how much dense geometry contributes to planning, we varied only the geometric supervision in the full streaming DVGT-2. All variants used the same architecture, initialization, data, and planning loss. Each skipped the 160K-step geometry pretraining stage and retained its assigned losses throughout 80K steps of streaming training and 40K steps of NAVSIM fine-tuning. Joint pointmap and ego-pose supervision achieved the highest PDMS (90.4), compared with 88.6 for depth alone, 89.1 for ego-pose alone, and 72.6 for planning supervision alone.

Geometric supervisionNAVSIM v1 navtest PDMS ↑
None (planning only)72.6
Depth only88.6
Ego-pose only89.1
Dense Geometry90.4

R2. Comparison under comparable input and training settings (W2, Q2).

VLA, WAM, and VGA models differ in architecture, making it difficult to match model size, training budgets, and camera inputs across methods. Some WAMs use large video backbones such as Wan pretrained on extensive video data, and some use a single view input to limit computation. VGA’s support for different camera configurations allows it to train across driving datasets and process multiple views efficiently.

For a controlled comparison, we pretrained DVGT-2 on OpenScene and fine-tuned it on NAVSIM with a total training budget comparable to baselines. Using the same three front views as both baselines, DVGT-2 achieved 90.2 PDMS, compared with 89.1 for AutoVLA and 89.2 for DriveDreamer-Policy. DVGT-2 also used fewer parameters and had lower inference latency.

MethodParametersPDMSLatency (ms)
DVGT-21.8B90.2260
AutoVLA3B89.112171
DriveDreamer-Policy3.3B89.2465
AuthorsOpenReview ↗

Author Response - Part 2/2

R3. Long-sequence PDMS evaluation in Table 6 (W2, Q3).

Table 6 evaluates the role of each streaming component during continuous driving. We cannot perform this ablation on the official NAVSIM navtest split because each scenario provides only four input frames, which are too few to assess how these components affect performance over long sequences. For example, each scenario ends before the history exceeds the default cache window and the cache has to be reset, or the model reaches temporal positions beyond those seen during training. We therefore used 200 continuous sequences from the OpenScene test split, each longer than 200 frames. We scored every frame with four seconds of future data (eight frames at 2 Hz). At each frame, the NAVSIM v1 PDM scorer evaluated the predicted trajectory from the recorded ego state using the corresponding map and surrounding traffic. Table 6 reports the mean PDMS over all scored frames across the 200 sequences, using the same frames for every variant.

R4. Streaming ablations with cache resets (W2, Q3).

We cannot run the full ablation in Table 6 without cache resets because rows c and d cache the entire history and would run out of GPU memory before completing the long sequences. Even though, it still allows us to isolate the effects of streaming components. Rows c and d use the same memory-triggered reset policy, so their comparison isolates the effect of local-coordinate reconstruction. Rows e and f both run without resets, so their comparison isolates the effect of relative temporal encoding. For rows d and e, avoiding cache-resets is itself a benefit of the fixed window. Row d must periodically clear its cache to complete a long sequence, losing recent context at each reset. Row e keeps memory bounded while retaining recent context throughout the sequence. The d–e comparison therefore captures the overall benefit of the fixed window during continuous streaming, including its ability to avoid this loss of context (cache-reset).

R5. Recording-level separation (Q4).

All OpenScene data used for pretraining came from the OpenScene trainval split. We evaluated on NAVSIM navtest, which is drawn from the OpenScene test split and shares no driving logs with trainval.

R6. Reactive planning on NAVSIM navhard (Q4).

We evaluated the DVGT-2 on NAVSIM v2.2 navhard_two_stage, using the official two-stage reactive protocol. The results validated the effectiveness of DVGT-2 under reactive planning conditions.

MethodCombined EPDMS ↑
DriveVLA-W024.4
DriveLaW30.6
DVGT-232.8

Dear Reviewer atTc,

Thank you for your thoughtful review and constructive suggestions, which have significantly improved the quality and clarity of our paper. We sincerely hope that our rebuttal has fully addressed all of your concerns. As the discussion period is ending soon, if you need any further clarification or additional information, please do not hesitate to contact us. We truly appreciate your time and effort devoted to reviewing our work.

Best regards, Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.