acceptodds
Withdrawn from ICLR 2027

VGGTWorld-VLA: Intent-Conditioned 3D World Evolution for Autonomous Driving

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

VGGT provides a strong foundation for geometry-centric world models by recovering unified 3D scene geometry from visual observations. Although recent extensions enable temporal 3D prediction, their future evolution remains weakly conditioned on driving intentions and actions, limiting their ability to model alternative action-dependent futures. We propose VGGTWorld-VLA, an intention-conditioned extension of VGGT-World for controllable 3D world evolution in autonomous driving. First, we introduce an action–semantic conditioning mechanism that injects complementary driving semantics and ego-motion representations into the future-token stream, enabling different future geometry predictions for the same observed scene under alternative ego actions. Second, we develop a geometry–language–action bridge that adapts historical geometry, VLA semantic features, and maneuver and trajectory representations for joint conditioning of future geometry prediction. We evaluate future geometry prediction on NAVSIM, while conditioning ablations further examine the contributions of semantic and action information. Compared with the baseline, our method demonstrates competitive geometry prediction performance. Ablation studies further support the effectiveness of semantic and action conditioning. These results demonstrate the potential of semantic and action conditioning for controllable VGGT-based world prediction in autonomous driving.

Primary area:
applications to robotics, autonomy, planning
Keywords:
3D World Models, Autonomous Driving, Depth Forecasting

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_6vJXOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper extends VGGT-World with action and semantic conditioning. Two historical geometry frames, supplied trajectories and derived maneuver categories, and Qwen2.5-VL features from three camera views and text prompts condition flow-based prediction of geometry tokens at 0.5 and 1.0 seconds. A VGGT decoder produces future depth and 3D structure. On 3,000 maneuver-balanced NAVSIM clips, evaluated against VGGT-derived pseudo-ground truth, RMSE decreases by 13.7% and δ1 increases by 4.11 percentage points over VGGT-World, while AbsRel slightly worsens. Validation is limited to offline conditional geometry prediction; planning and closed-loop results are not reported.

Critical strengths
  1. Action-conditioned geometry prediction is well motivated. Predicting the spatial consequences of supplied actions is a useful direction for candidate-action evaluation. Preserving individual waypoints alongside a maneuver category also has empirical support: the joint representation improves most decoded geometry metrics in Table 2.

  2. Prediction gains and action utilization have complementary support. RMSE decreases by 15.7% against the NAVSIM-adapted VGGT-World-N baseline (Table 1), partially addressing gains from domain adaptation alone. Controlled interventions with fixed history, semantics, and initial noise, together with the additional multi-scene visualizations, support actual use of action conditions.

Critical weaknesses
  1. Evaluation-data and condition provenance remain unclear. Neither the main paper nor the supplement specifies the log- or scene-level training/evaluation split or the relationship between the 3,000 evaluation clips and training data. It also remains unclear whether the “supplied action” is a recorded trajectory or a model prediction, and how system/user prompts are constructed. These details directly affect the interpretation of generalization and the conditional-prediction task. Recorded trajectories are valid conditions, but their use should be explicit.

  2. Geometric control and the semantic contribution require more targeted validation. Table 3 and Figures A2–A4 support action responsiveness, but do not quantify trajectory-consistent camera motion or gains over static reprojection. Table 4 shows consistent but modest semantic-branch gains, including a 0.84% RMSE reduction and a 0.73% $\delta$1 increase. Although C.3 matches the training budget, this branch also introduces three camera views, whereas the geometry branch uses single-camera history. The additional latent metrics do not isolate semantic understanding from wider visual coverage or feature capacity.

Questions for authors
  1. Please specify the log-/scene-level training and evaluation split and how the 3,000 evaluation clips are separated from training. Are the supplied trajectories recorded, VLA-predicted, or obtained elsewhere? How are system/user prompts constructed, and what information do they use?

  2. Can you quantify consistency between supplied trajectories and predicted camera motion, or compare against static reprojection on mutually visible regions using the same history and actions? This would clarify the control capability illustrated by the additional qualitative results.

  3. With matched three-camera inputs, training budget, and adapter configuration, do the Table 4 gains persist when ordinary visual features replace VLM features? How large are these differences relative to training or sampling variability?

Reviewer_CqUXOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper extends VGGT-World by incorporating ego-motion and VLM semantic features to improve short-horizon 3D geometry prediction and enable action-conditioned future prediction. While the authors claim that VGGTWorld-VLA can capture driving intentions, model complex traffic interactions, and predict different future 3D worlds under alternative ego actions, the current experiments mainly demonstrate improvements in short-horizon geometry prediction. There remains a gap between the claimed capabilities and the experimental evidence, particularly regarding controllable and long-horizon 3D world evolution.

Critical strengths

-The proposed method is simple and straightforward. Ego-motion and semantic features are encoded separately and injected into the flow-based world model, leading to improved geometry prediction performance over VGGT-World.

-The ablation studies in Tables 2–4 demonstrate the effectiveness of different ego-motion representations and the additional benefits of semantic conditioning for geometry prediction.

Critical weaknesses

-The authors claim that a central contribution of this paper is the ability to generate different future 3D worlds conditioned on alternative ego actions. However, the existing experiments (e.g., Table 3) primarily demonstrate the model's sensitivity to correct or incorrect action inputs, rather than the accuracy or physical plausibility of the predicted futures under different actions. I strongly encourage the authors to provide dedicated quantitative and qualitative evaluations of this capability, ideally with appropriate ground-truth future states or reliable evaluation protocols for alternative ego actions. Moreover, the current visualizations in Figures 5 and A2 are insufficient to substantiate this central claim. Additional long-horizon action-conditioned prediction results would be more convincing.

-The current evaluation relies on VGGT-generated pseudo GT. I understand that NAVSIM lacks dense depth GT and that VGGT-generated pseudo GT is a reasonable alternative, the evaluation should distinguish between recorded-future and counterfactual-future prediction. For recorded futures, more reliable geometric references or independent consistency metrics could be considered. For counterfactual futures under alternative ego actions, where ground truth is unavailable, dedicated evaluation protocols are needed to assess action consistency, geometric accuracy, and physical plausibility. Overall, these two scenarios require different evaluation criteria to adequately support the claimed capability of controllable 3D world evolution.

Questions for authors

My main concerns are outlined in the Critical Weaknesses section, as they directly relate to the paper's central contributions and require stronger experimental evidence. I also have the following questions and suggestions need to be clarify: -The authors adopt high-level VLM embeddings as semantic features to guide fine-grained pixel-level or point-level geometry prediction. It remains unclear whether these embeddings preserve sufficient spatial details and provide additional information beyond dense semantic representations, such as pixel-level semantic segmentation. In particular, do they contribute meaningful information about actor dynamics, traffic interactions, or driving intentions? Further justification and experimental evidence would strengthen this design choice.

-The qualitative results in Figures 5 and A2 cover only a 1-second prediction horizon and mainly illustrate static scene structures. Longer-horizon predictions involving moving actors and complex traffic interactions would provide stronger evidence of the proposed world evolution capability.

-The evaluation focuses heavily on depth prediction accuracy, while other aspects of 3D geometry, such as point-cloud reconstruction and camera pose prediction, receive limited attention. More comprehensive quantitative evaluations and visualizations of the predicted 3D geometry would better support the paper's claims.

-The visualizations in Figure 1 appear to be AI-generated. I strongly recommend replacing them with actual visualization results produced by the proposed method, rather than AI-generated illustrations, to better demonstrate the model's capabilities.

Reviewer_EaRzOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes VGGTWorld-VLA, which extends VGGT-World with action and VLM-derived semantic conditioning for future 3D geometry prediction. It injects encoded ego trajectories and semantic features into a flow-based world model to predict future geometry tokens and depth. Experiments on a 3,000-clip NAVSIM subset show improvements over VGGT-World on some metrics, with ablations examining action representations and semantic conditioning.

Critical strengths

The paper investigates action-conditioned future geometry prediction and includes controlled action-input interventions. The results suggest that the predicted geometry is sensitive to the supplied action conditions.

Critical weaknesses
  1. Unclear motivation and task positioning. Despite the VLA framing, the primary task is future geometry prediction conditioned on a supplied trajectory and VLM-derived semantic features. The paper does not demonstrate improved action selection or downstream driving performance, leaving the practical motivation and value of the proposed framework insufficiently established.

  2. Limited novelty and comparison. The method primarily extends VGGT-World by injecting action and semantic conditions through cross-attention. The paper does not sufficiently distinguish this design from existing action-conditioned world models or establish the specific advantages of the proposed architecture.

  3. Insufficient evidence of improved geometric accuracy. Evaluation relies on pseudo-ground-truth generated by a pretrained VGGT model rather than ground-truth depth. Moreover, RMSE and δ1 improve while AbsRel deteriorates. These results do not establish an overall improvement in actual geometric prediction quality.

  4. Unclear presentation and claims. The relationship between the VLA component, action conditioning, and the claimed world-action contribution is not clearly articulated. The paper would benefit from a more precise problem formulation and a clearer explanation of what capabilities are demonstrated versus what remains future work.

Questions for authors
  1. What concrete downstream benefit does the proposed action-conditioned geometry prediction provide beyond improved agreement with VGGT-generated pseudo-labels? Can the authors demonstrate gains in planning or action selection?

  2. How does the proposed conditioning mechanism differ substantively from existing action-conditioned world models, and what evidence isolates the contribution of VLM-derived semantics from that of trajectory conditioning?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.