acceptodds
Under review as a conference paper at ICLR 2027

DVGT-World: Extending Vision-Geometry Action Paradigms To 3D World Models For Autonomous Driving

13 views

Abstract

Autonomous driving requires not only understanding the current 3D scene but also predicting how it will evolve. Recent Vision-Geometry-Action (VGA) models have advanced end-to-end driving through explicit 3D geometry reconstruction. However, most VGA methods focus on reconstructing currently observed scenes, lacking explicit modeling of future 3D dynamics. While World-Action Models (WAMs) forecast scene evolution, their predictions in 2D image or latent spaces lack explicit 3D geometric supervision. To address this, we present DVGT-World, a framework that extends the VGA paradigm into a predictive 3D world model. By jointly supervising current geometry reconstruction and future geometry prediction, DVGT-World learns representations that capture both spatial structure and scene evolution, providing a geometric foundation for downstream planning. Specifically, we introduce learnable future queries conditioned on the target horizon into the geometry transformer to predict future dense pointmaps and ego poses from historical multi-view observations. After geometry pretraining, we fine-tune the model for planning, with future prediction disabled for inference efficiency. Experiments demonstrate accurate multi-view future geometry prediction and strong closed-loop planning performance, achieving 93.3 PDMS on NAVSIM v1 and 90.7 EPDMS on NAVSIM v2, which validates the benefit of modeling 3D scene evolution through future geometry prediction for downstream planning.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Autonomous Driving

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_Q9chOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DVGT-World, which extends the streaming Vision-Geometry-Action (VGA) framework with horizon-conditioned future geometry prediction. The model jointly learns current multi-view geometry reconstruction and future pointmap and ego-pose prediction during pretraining, then fine-tunes for trajectory planning on NAVSIM. The future prediction branch is disabled during planning inference to avoid additional computational cost.

Critical strengths
  1. The paper presents a straightforward extension of the VGA paradigm by explicitly supervising future 3D geometry. The future prediction branch can be disabled during planning inference, making the approach practical without incurring additional inference cost from future geometry prediction.

  2. The experiments cover planning performance, future geometry prediction, prediction horizons, and inference efficiency. The ablation study shows a consistent planning benefit from future geometry supervision, with PDMS increasing from 89.9 to 93.0 on NAVSIM v1.

Critical weaknesses
  1. The contribution of the scorer is not isolated. The scorer is trained directly on NAVSIM driving metrics and selects the final trajectory from 64 candidates, so it can materially affect the reported planning scores. However, the paper does not compare against a standard planning head without a scorer. It is therefore unclear how much of the performance gain comes from future geometry pretraining versus the scoring module.

  2. Insufficient comparison with trajectory scoring methods. The paper lacks a controlled comparison with dedicated trajectory scoring approaches. In particular, the planning tables should include relevant methods such as DrivoR, which explicitly scores candidate trajectories. Comparisons under a common candidate generation and evaluation setup would better establish the effectiveness of the proposed planning pipeline.

  3. Insufficient evidence for representation effectiveness. The planning ablations suggest that future geometry supervision benefits downstream planning, but the paper does not directly evaluate representation quality or transferability against alternative pretraining strategies under a common planning setup.

Questions for authors
  1. Can the authors provide an ablation using a standard planning head without a scorer, keeping the backbone, candidate generation, and training setup fixed? This would help isolate the contribution of future geometry supervision from that of trajectory scoring.

  2. Can the authors compare their scoring approach with dedicated trajectory scoring methods under a common candidate set and evaluation protocol? In particular, how much of the improvement comes from the proposed representation versus the scoring and selection strategy?

4 comments
AuthorsOpenReview ↗

Author Response - Part 1/2

We thank the reviewer for the constructive suggestions. Sorry for the late reply, as we spent considerable time completing the dense training experiments. We first clarify a correction to Table 3 and then respond point by point.

Correction to Table 3. The submitted text stated that all Table 3 variants followed the same training protocol, which was inaccurate. The No Future baseline used the diffusion planner of DVGT-2, while all future-supervised variants used our scorer-based planner. We have retrained the No Future baseline with the same scorer-based planner under the full Table 1 configuration. Under this matched setting, future geometry supervision improves PDMS from 92.29 to 93.28 (+0.99). This matched comparison still supports our central claim that future 3D dense geometry prediction benefits downstream planning. The prediction-range variants all use the scorer-based planner, so their conclusions are unaffected.

R1. Contributions of future geometry supervision and the scorer head (W1, Q1).

To isolate the contributions of future geometry prediction and the scorer head, we trained three variants with the same initialization, training data, and optimization budget. On NAVSIM v1 navtest, replacing the diffusion planner with the scorer-based planner improved PDMS from 90.29 to 92.29 (+2.00) with future prediction disabled. With the scorer-based planner held fixed, adding future geometry prediction further improved PDMS from 92.29 to 93.28 (+0.99), supporting our central claim that predicting future dense geometry benefits downstream planning. The fourth variant would test whether this gain also holds for the diffusion planner. We could not complete that run within the available GPU time during rebuttal and will report it in future work.

PlannerFuture supervisionNAVSIM v1 navtest PDMS ↑
Diffusion plannerNo Future90.29
Scorer-based plannerNo Future92.29
Scorer-based plannerDense geometry93.28

R2. Comparison with DrivoR and contribution of future geometry pretraining (W2, Q2).

We compared DVGT-World with scorer-based baselines, including DrivoR, using the standard NAVSIM navtrain split for planning training. Each method generates its own candidate trajectories and selects trajectories using its own scorer, so we report results as an end-to-end comparison. On navtest, DVGT-World achieved 93.3 PDMS, outperforming SparseDriveV2 and trailing DrivoR (93.7) by 0.4 points. This small gap may be partly due to DrivoR’s higher camera input resolution (1148 × 672 vs. 512 × 288) and our limited exploration of scorer design, as this work focuses on geometry representations and future supervision rather than the scorer design. Our performance remains competitive with state-of-the-art planners on a benchmark where the human reference achieves 94.8 PDMS. Meanwhile, DVGT-World achieved higher NC and TTC scores than DrivoR, indicating fewer at-fault collisions and lower near-term collision risk, consistent with our motivation to learn spatially aware representations through future geometry prediction for robust planning. We have included these results in the revised paper.

The scorer design is not the contribution of this paper. DrivoR improves the planner through register-based token compression and disentangled scoring, while our contribution is the pretraining objective, so the two directions are orthogonal. To assess the contribution of our pretrained representation, we kept the generator and scorer architectures, selection rule, and training budget fixed. As shown in R1, adding future geometry prediction during pretraining improved PDMS from 92.29 to 93.28 under the same planning setup. This supports our central claim that future 3D dense geometry prediction benefits downstream planning.

MethodNC ↑DAC ↑TTC ↑Comf. ↑EP ↑PDMS ↑
SparseDriveV298.598.495.099.988.692.0
DrivoR99.098.996.710090.093.7
DVGT-World99.298.196.810089.993.3
AuthorsOpenReview ↗

Author Response - Part 2/2

R3. Comparison of different pretrained representations (W3).

To validate the effectiveness of our representation, we compared four variants using the same training data, optimization budget, and scorer-based planner. All variants retained current geometry reconstruction and differed only in future supervision. Representations pretrained to predict future dense geometry transferred best to planning, achieving 93.28 PDMS on NAVSIM v1 navtest, 0.73 points higher than future depth prediction and 0.80 points higher than future visual latent prediction. The visual latent variant serves as a controlled counterpart of world-action models that predict future video latents, such as Fast-WAM.

Future supervisionNAVSIM v1 navtest PDMS ↑
No Future92.29
Depth92.55
Visual latent92.48
Dense geometry93.28

Thank you for the detailed rebuttal and additional experiments.

I appreciate the authors' efforts to address my concerns. The new controlled experiments provide stronger evidence for the effectiveness of future geometry supervision, and the revisions have clarified the experimental setup and conclusions.

Overall, I believe my main concerns have been adequately addressed, and I am updating my rating.

AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewer Q9ch,

Thanks for taking the time to review our paper and rebuttal.

Your suggestions helped us improve the experiments and make the paper clearer. We are glad that the additional experiments and revisions have addressed your main concerns, and we appreciate your constructive feedback and positive assessment of our work.

Best regards, The Authors

Reviewer_iTy2OpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

DVGT-World extends DVGT-2 with horizon-conditioned future queries that predict dense multi-view pointmaps and ego poses from historical images. Joint reconstruction and prediction pretraining on OpenScene is followed by planning fine-tuning on NAVSIM. Causal attention allows future queries to be removed during planning. The paper reports 93.3 PDMS on NAVSIM v1 and 90.7 EPDMS on v2, alongside geometry prediction over a four-second horizon.

Critical strengths
  1. Meaningful planning gains supported by a controlled ablation. Under the same training protocol, future supervision improves PDMS from 89.9 to 93.0, with improvements in collision-related metrics and route progress (Table 3). The main model also outperforms the compared methods on both NAVSIM versions (Tables 1–2). This provides substantive evidence that predictive pretraining benefits downstream planning.

  2. An efficient mechanism for transferring predictive supervision. Future queries access observed features without affecting their forward computation, allowing prediction to be disabled during planning (Section 3.4). Table 6 reports 222 ms latency with future prediction disabled versus 414 ms enabled. This makes the approach practically attractive: planning benefits from predictive training without requiring future geometry generation at deployment.

Critical weaknesses
  1. The contribution of dense future geometry is not isolated from future ego-motion supervision. The pretraining objective jointly adds future pointmap and ego-pose losses, while Table 3 varies them together (Section 3.5). Future ego-pose prediction already provides supervision closely related to trajectory planning. Without pose-only and pointmap-only controls, the results establish the value of joint future supervision but do not determine whether modeling dense scene evolution drives the planning improvement. This affects the central technical claim.
  2. The geometry evaluation provides limited evidence of dynamic-scene understanding. Tables 4–5 report aggregate geometry metrics without separating moving objects from static background. Nearest-neighbor pointmap distances can obscure object-motion errors, and supervision derives from MoGe-2 depth estimates (Section 4.1; Appendix B). Additionally, pose AUC measures rotation and translation-direction errors rather than translation magnitude. Consequently, the reported results do not fully establish accurate prediction of dynamic objects and metric ego motion.
Questions for authors
  1. Attribution: With current reconstruction retained, how do future-pose-only, future-pointmap-only, and joint supervision compare under the Table 3 training protocol?
  2. Dynamic prediction: Can the Table 4 comparison be broken down into moving-object and static-background errors? Does the advantage over GT-Pose + DVGT-2 persist specifically on moving objects?
  3. Metric ego motion: What are the future translation errors in meters at 1–4 seconds, complementing the angular AUC in Table 5?
3 comments
AuthorsOpenReview ↗

Author Response

We thank the reviewer for the positive assessment and constructive suggestions. Sorry for the late reply, as we spent considerable time completing the dense training experiments. We first clarify a correction to Table 3 and then respond point by point.

Correction to Table 3. The submitted text stated that all Table 3 variants followed the same training protocol, which was inaccurate. The No Future baseline used the diffusion planner of DVGT-2, while all future-supervised variants used our scorer-based planner. We have retrained the No Future baseline with the same scorer-based planner under the full Table 1 configuration. Under this matched setting, future geometry supervision improves PDMS from 92.29 to 93.28 (+0.99). This matched comparison still supports our central claim that future 3D dense geometry prediction benefits downstream planning. The prediction-range variants all use the scorer-based planner, so their conclusions are unaffected.

R1. Contribution of future pointmap and ego-pose supervision (W1, Q1).

We trained all four variants under the full Table 1 configuration with the same initialization, scorer-based planner, and training budget. All retained current geometry reconstruction and differed only in future supervision. On NAVSIM v1 navtest, pointmap-only and ego-pose-only supervision achieved 92.72 and 92.78 PDMS, respectively, compared with 92.29 for No Future baseline. Adding future pointmap prediction to ego-pose supervision improved PDMS by a further 0.50 points, giving joint supervision the highest score (93.28). This indicates that modeling dense scene geometric evolution contributes to planning beyond future ego-motion supervision.

Future supervisionNAVSIM v1 navtest PDMS ↑
No Future92.29
Ego-pose only92.78
Pointmap only92.72
Pointmap + ego-pose93.28

R2. Future geometry prediction on moving objects and static background (W2, Q2).

We evaluated front-view pointmaps separately for moving objects and static background at 1–4 s on NAVSIM v1 navtest, using the same samples and semantic masks for all methods. For moving objects, we compared predicted points directly with target points at the objects’ ground-truth future positions.

We compared DVGT-World with two baselines. The first compensates only for ego motion by transforming DVGT-2’s current pointmap into the future ego frame using the ground-truth ego pose (DVGT-2 + GT ego pose). The second also compensates for object motion by moving each tracked object’s points according to its ground-truth 3D box motion (DVGT-2 + GT ego pose + GT 3D box motion). DVGT-World achieved lower Acc and Comp errors on moving objects at all four horizons, including when the baseline used ground-truth 3D boxes to compensate for object motion. At 4 s, its Acc / Comp on moving objects was 1.02 / 2.15 m, compared with 6.24 / 6.44 m for the baseline that compensates for both ego and object motion.

Each cell reports Acc / Comp in meters (lower is better).

MethodMoving 1 sMoving 2 sMoving 3 sMoving 4 sStatic 1 sStatic 2 sStatic 3 sStatic 4 s
DVGT-2 + GT ego pose1.97 / 1.963.79 / 3.525.25 / 4.897.00 / 6.561.05 / 0.891.99 / 1.483.10 / 2.184.08 / 2.87
DVGT-2 + GT ego pose + GT 3D box motion1.81 / 1.853.35 / 3.424.63 / 4.766.24 / 6.441.01 / 0.891.91 / 1.473.05 / 2.163.96 / 2.83
DVGT-World0.50 / 0.930.74 / 1.490.82 / 1.651.02 / 2.150.44 / 0.720.52 / 0.970.60 / 1.220.67 / 1.49

R3. Future ego-translation error in meters (W2, Q3).

To complement the AUC@30 metric in Table 5, which measures angular errors in rotation and translation direction, we report the L2 distance between predicted and ground-truth ego translations at four future horizons on NAVSIM v1 navtest. The translation error of future pose prediction increases from 0.17 m at 1 s to 1.63 m at 4 s.

Future horizon1 s2 s3 s4 s
Ego-translation L2 error (m) ↓0.170.500.991.63

Thank you for the detailed rebuttal and additional experiments. I think your complementary experiments provide more evidence to strengthen your paper.

I hope you could update these experiments in your final paper.

Dear Reviewer iTy2,

Thank you for your positive feedback and for recognizing the value of our additional experiments. We have incorporated these complementary experiments and the corresponding discussions into the revised version of our paper. We sincerely appreciate your constructive comments, which have helped us strengthen this work.

Best regards, The Authors

Reviewer_AZ4FOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DVGT-World extends DVGT-2 with horizon-conditioned future queries to predict future dense pointmaps and ego-poses. The model is pretrained with current and future geometry supervision, then fine-tuned for planning with the future branch disabled. Experiments report 93.3 PDMS on NAVSIM v1 and 90.7 EPDMS on NAVSIM v2, together with future geometry evaluations and planning ablations.

Critical strengths

The problem is relevant and the design is simple. Extending driving geometry pretraining from reconstruction to future prediction is a reasonable way to improve planning representations. The causal attention design allows the future branch to be removed during planning, avoiding its additional inference cost.

The internal ablation supports the usefulness of future supervision. Under the stated matched training protocol, Table 3 improves PDMS from 89.9 without future supervision to 93.0 with supervision spanning eight future frames. The paper also evaluates future geometry quality and the effect of the prediction range.

Critical weaknesses

The benefit of future pointmap prediction is not isolated. Equation (2) adds both future pointmap and future ego-pose losses. Future ego translation and rotation closely overlap with the planning target, so the gains could come from ego-motion prediction rather than pointmap prediction. Table 3 removes both objectives together. Separate pose-only, pointmap-only, and joint ablations are needed to establish the contribution of future scene geometry.

The added value over existing future-prediction methods is unclear. Fast-WAM already uses future prediction during training and attention masks that allow the future branch to be removed at inference. This closely related design is not discussed. Comparisons with future-depth or visual-latent prediction under the same backbone and training budget are also missing. The experiments show that future supervision helps, but do not establish the advantage of choosing dense 3D geometry as the prediction target.

Strong scoring-based baselines and planning details are missing. DrivoR reports 93.1 PDMS with train-only training and 93.7 with train+val, compared with 93.3 here, but is omitted. The planning and selection scheme also differs from DVGT-2, so the main-table gain cannot be attributed solely to future geometry pretraining. Table 3 controls for these changes, but uses a smaller training configuration. The generator and scorer architectures are insufficiently specified, including whether DVGT-2's diffusion decoder is retained.

Questions for authors

Can you report future-pose-only, future-pointmap-only, and joint supervision under the same initialization, planning configuration, and training budget?

How does future pointmap supervision compare with future depth or visual-latent prediction under matched settings? What distinguishes the method from Fast-WAM?

Please specify the generator and scorer architectures, including whether diffusion is retained. Are the planning settings identical across Table 3 variants? Can you include DrivoR and a full-configuration No Future baseline?

Can you report future geometry errors separately for static and dynamic regions, with ground-truth comparisons of moving agents' future positions?

3 comments
AuthorsOpenReview ↗

Author Response - Part 1/2

We thank the reviewer for the constructive suggestions. Sorry for the late reply, as we spent considerable time completing the additional experiments. We first clarify a correction to Table 3 that affects several questions, and then respond point by point.

Correction to Table 3. The submitted text stated that all Table 3 variants followed the same training protocol, which was inaccurate. The No Future baseline used the diffusion planner of DVGT-2, while all future-supervised variants used our scorer-based planner. We have retrained the No Future baseline with the same scorer-based planner under the full Table 1 configuration. Under this matched setting, future geometry supervision improves PDMS from 92.29 to 93.28 (+0.99). This matched comparison still supports our central claim that future 3D dense geometry prediction benefits downstream planning. The prediction-range variants all use the scorer-based planner, so their conclusions are unaffected.

R1. Contribution of future pointmap and ego-pose supervision (W1, Q1).

To isolate the contributions of future pointmap and ego-pose prediction, we trained four variants with the same initialization, planning configuration, and training budget, which vary only the future supervision. On NAVSIM v1 navtest, predicting future pointmaps or ego poses alone improved PDMS by 0.43 and 0.49 points over the No Future baseline, respectively. Adding future pointmap prediction on top of ego-pose supervision brought a further 0.50-point gain, and joint prediction achieved the highest PDMS (93.28). This indicates that future scene geometry provides planning-relevant information beyond future ego motion.

Future supervisionNAVSIM v1 navtest PDMS ↑
No Future92.29
Ego-pose only92.78
Pointmap only92.72
Pointmap + ego-pose93.28

R2. Future prediction targets and comparison with Fast-WAM (W2, Q2).

Fast-WAM and DVGT-World both use future prediction during training and omit the future branch at inference. The core difference is that Fast-WAM predicts future video latents that capture 2D appearance, while DVGT-World predicts future 3D dense geometry through pointmaps and ego-poses. Since vehicles move through a 3D world, we expect predicting future in 3D scene geometry to better support trajectory planning than predicting future in 2D pixel space. To validate this, we replaced the future dense geometry target with future depth and with future visual latents, the latter serving as a controlled counterpart of the Fast-WAM target, while keeping the backbone, training budget, and planner fixed. On NAVSIM v1 navtest, future dense geometry prediction achieved the highest PDMS (93.28), exceeding depth prediction by 0.73 points and visual latent prediction by 0.80 points. We have added this discussion to the revised paper.

Future prediction targetNAVSIM v1 navtest PDMS ↑
Depth92.55
Visual latent92.48
Dense geometry93.28
AuthorsOpenReview ↗

Author Response - Part 2/2

R3. Planning architecture and comparison with DrivoR (W3, Q3).

We apologize for the confusion about the planning setup in Table 3. In DVGT-World, the generator replaces DVGT-2’s diffusion decoder with a four-layer transformer decoder that predicts 64 candidate trajectories from scene features. The scorer uses a separate four-layer transformer decoder to predict NAVSIM driving metrics for each candidate. We select the trajectory with the highest combined score as the final prediction.

We also compared DVGT-World with scorer-based baselines, including DrivoR, using the standard NAVSIM navtrain split for planning training. On navtest, DVGT-World achieved 93.3 PDMS, outperforming SparseDriveV2 and trailing DrivoR (93.7) by 0.4 points. Notably, we achieve comparable performance despite using over 5× fewer input pixels than DrivoR (512 × 288 vs. 1148 × 672) and a simpler scorer. Our scorer is kept simple by design, as this work focuses on geometry representations and future supervision. Our performance remains competitive with state-of-the-art planners on a benchmark where the human reference achieves 94.8 PDMS. Meanwhile, DVGT-World achieved higher NC and TTC scores than DrivoR, indicating fewer at-fault collisions and lower near-term collision risk, consistent with our motivation to learn spatially aware representations through future geometry prediction for robust planning. We have included these results in the revised paper.

MethodNC ↑DAC ↑TTC ↑Comf. ↑EP ↑PDMS ↑
SparseDriveV298.598.495.099.988.692.0
DrivoR99.098.996.710090.093.7
DVGT-World99.298.196.810089.993.3

R4. Future geometry prediction in moving and static regions (Q4).

We evaluated front-view pointmaps at 1–4 s on NAVSIM v1 navtest, computing Acc and Comp separately for moving and static regions using the same semantic masks for all methods. In moving regions, we compared predicted points directly with target points on moving objects at their ground-truth future positions.

We compared DVGT-World with two baselines. The first compensates only for ego motion by transforming DVGT-2’s current pointmap into the future ego frame using the ground-truth ego pose. The second also compensates for object motion by moving each tracked object’s points according to its ground-truth 3D box motion. DVGT-World achieved lower errors than both baselines in moving and static regions at every horizon. At 4 s, its Acc / Comp on moving objects was 1.02 / 2.15 m, compared with 6.24 / 6.44 m for the baseline that compensates for both ego and object motion.

Each cell reports Acc / Comp in meters (lower is better).

MethodMoving 1 sMoving 2 sMoving 3 sMoving 4 sStatic 1 sStatic 2 sStatic 3 sStatic 4 s
DVGT-2 + GT ego pose1.97 / 1.963.79 / 3.525.25 / 4.897.00 / 6.561.05 / 0.891.99 / 1.483.10 / 2.184.08 / 2.87
DVGT-2 + GT ego pose + GT 3D box motion1.81 / 1.853.35 / 3.424.63 / 4.766.24 / 6.441.01 / 0.891.91 / 1.473.05 / 2.163.96 / 2.83
DVGT-World0.50 / 0.930.74 / 1.490.82 / 1.651.02 / 2.150.44 / 0.720.52 / 0.970.60 / 1.220.67 / 1.49

Thank you for the response and for taking the time to run the additional experiments. The ablations address my main concern about whether future pointmaps help beyond ego-pose supervision. I'm still unsure whether the depth and visual-latent baselines used the same ego-pose supervision. I'll raise my score to 3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.