acceptodds
Withdrawn from ICLR 2027

VLG-Drive: Unifying Vision-Language and Geometric Priors for Autonomous Driving

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

A driving planner needs both scene semantics, which vision-language models (VLMs) provide, and spatial structure, which geometric foundation models recover from visual history. We present VLG-Drive, which keeps both models frozen and trains only a fusion interface and a planner from trajectory labels. At each camera-grid cell, the interface fuses the paired VLM and geometry features into one joint token through a geometry-anchored residual, and a command-conditioned planner reads these tokens to predict the ego trajectory. VLG-Drive achieves 31.78 EPDMS on NavHard under perturbed initial states, the highest among the compared methods, and 88.27 PDMS on NavTest from cameras alone. Together with consistent gains over independent VLM and geometry streams, these results show that planning directly from the fused features of vision-language and geometric foundation models opens a new direction for building driving planners.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Autonmous driving, 3D vision

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_e2EmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces VLG-Drive, a driving planner built on two frozen foundation models (2B Qwen3-VL and VGGT-Ω). Features from both are pooled onto a shared grid per camera across three cameras and tagged with Plücker encodings. A zero-initialized residual MLP then fuses each VLM–geometry pair into a single geometry-anchored token, which replaces both dense streams in the planner memory. A command-conditioned query decoder regresses eight incremental ego poses. Training is restricted to the interface and planner. The method reaches 31.78 EPDMS on NavHard, the best among compared methods, and 88.27 PDMS on NavTest with cameras only. Ablation studies examine: fusion vs. independent streams vs. no-injection carriers, reduced label fractions, a second geometry backbone (DVGT-2), feature-level selection, geometry history length, attention and intervention analyses.

Critical strengths
  • The interface is minimal and well motivated. Exploiting the known pixel-level correspondence instead of letting decoder attention discover it is a clean idea, and zero-initialization from the geometry anchor is a sensible design choice.
  • The no-injection carrier separates fusion from memory compaction. The label-fraction study is matched to the main recipe.
  • The analysis goes beyond attention maps. Mean-feature interventions on top-attended and path-tube cells, with random controls, give causal evidence of where the planner reads.
  • The paper is very clearly written, and the implementation and protocol details are thorough.
Critical weaknesses
  • Confounded fusion study (Table 3). The headline row, geometry carrier with pointwise fusion, uses the main recipe. All other rows use the base recipe. This confounds both the fusion vs. independent streams comparison and the carrier comparison. A base-recipe geometry-carrier fusion row is needed.
  • No variance estimates. All results are single runs.
  • Hyperparameter selection on test benchmarks. History length (Table 8) and feature levels (Table 6) appear to be chosen from NavTest/NavHard scores. The reported best configuration is therefore partially selected on the evaluation sets. A held-out validation split should drive these choices.
  • Unclear driving supervision. Table 2 lists “None” for auxiliary supervision, yet the VLM is “driving-adapted”. The paper should state which model and data were used for this adaptation. In particular, it should say whether that data overlaps with nuPlan logs underlying NavTest/NavHard.
  • Limited novelty. Adding aligned geometry features to VLM visual tokens through an MLP or residual connector is already used. The main difference is decoding with a dedicated planner rather than the LLM. Claiming a new direction overstates this. The contribution is a well-executed combination rather than a new mechanism.
  • Missing computational cost analysis. Each planning step runs a 2B VLM plus VGGT-Ω over three cameras × three frames, but no latency, FLOPs, or memory figures are reported. Training on cached features hides this cost. Without these numbers, the practical appeal of “lightweight” is unclear.
Questions for authors
  • What is the driving-adapted Qwen3-VL? Is there any overlap with NAVSIM train or test logs?
  • Can you report geometry-carrier fusion under the base recipe, and seed variance for the main NavHard/NavTest numbers and the key ablations?
  • What are the end-to-end inference latency and GPU memory compared with DVGT-2 and DiffusionDrive?
  • How are Qwen3-VL image tokens, with dynamic resolution and patch merging, mapped onto the grid? How sensitive are results to grid resolution?
  • Have you run any closed-loop or reactive evaluation?
4 comments

Response to Reviewer e2Em (1/2)

We thank the reviewer for the thorough review. Closed-loop results (Table R1) are in our general response.

Q1. The driving-adapted Qwen3-VL and data overlap. It is the publicly released Stage-1 nuScenes checkpoint of UniDriveVLA (2B), fine-tuned from Qwen3-VL-2B-Instruct. According to its released training configuration, its data are driving QA and planning corpora built on nuScenes and on BDD-X, DRAMA, LingoQA, CODA-LM, Talk2Car, SUTD-TrafficQA, and MAPLM, plus a general VQA subset. None is derived from nuPlan, the source of the NAVSIM logs, so no NAVSIM training or test log reaches the VLM during its adaptation. We will state this in the paper.

Q2a. Geometry-carrier fusion under the base recipe. We trained this row (Table R2): 88.06 PDMS / 30.75 EPDMS, versus 87.12 / 28.69 for independent streams and 86.87 / 29.18 for the no-injection geometry carrier, which has the same memory layout as fusion. Under the main recipe, fusion is also ahead of independent streams at 10%, 25%, and 100% of the labels on both benchmarks.

Table R2. Fusion study, all rows on the base recipe (decoder width 512, 12,000 updates, same optimizer, schedule, and data). The geometry-carrier fusion row and the NavHard column are new.

Dense representationInteractionPDMSEPDMS
Independent streamsDecoder attention only87.1228.69
VLM carrierNo dense geometry injection85.2325.89
VLM carrierPointwise fusion87.7829.47
Geometry carrierNo dense VLM injection86.8729.18
Geometry carrierPointwise fusion88.0630.75

Q2b. Seed variance. Table R4: three training runs of the main model; every run is above every NavHard baseline. Closed-loop results include 95% bootstrap intervals (Table R1).

Table R4. Training runs of the main model.

RunPDMSEPDMS
Seed 0 (reported)88.2731.78
Seed 188.3531.81
Seed 0, second run88.2131.39
Mean ± s.d.88.28 ± 0.0731.66 ± 0.23

Q3. Latency, memory, FLOPs. VLG-Drive takes 1123 ms and 9.8 GiB per step against 425 ms and 11.4 GiB for DVGT-2 (Table R5). The 2B VLM dominates our cost (65 of 87 TFLOPs); the geometry-only variant takes 450 ms. On the training side, VLG-Drive trains 106.1M parameters in about 58 GPU-h including the one-time caches, while DVGT-2 reports about 15,000 GPU-h.

Table R5. Cost (one RTX 6000 Ada; HUGSIM input, three 1920×1080 cameras; FLOPs per planning step; †: reported by the authors).

MethodTotal paramsTrainableTraining (GPU-h)Latency (ms)Memory (GiB)TFLOPs
VLG-Drive3.38B106.1M29 (+29 once for caches)11239.887.0
Geometry only1.25B103.0M274505.321.7
VGGDrive9.39B8.48Bn/r302821.677.0
DVGT-21.71B1.71B≈15,000†42511.415.5

Response to Reviewer e2Em (2/2)

History length and feature levels. These two studies are not how the main configuration was tuned. The main model, final-layer fusion with three geometry timestamps, is the reference configuration on which every other experiment is built (the fusion study, reduced labels, the second backbone, the no-history control, calibration, and closed loop). The history-length and feature-level studies are follow-up analyses of this design: they ask how much visual history the geometry branch needs and whether fusing intermediate features adds information beyond the final layer.

NAVSIM provides no official validation split, and results are commonly reported on NavTest directly. To address the concern nonetheless, we evaluated both studies on a held-out split of 2,876 scenarios from NavTrain logs that no model trains on, scored with the NavTest PDMS protocol at the final checkpoint (Val in Tables R9 and R10). It agrees with the test benchmarks on the main findings: a second timestamp gives the largest improvement, and depth-paired multi-level fusion (LP-PAIR) improves on final-layer fusion, with the best held-out and NavTest PDMS. It differs in preferring two timestamps over three. Neither difference affects our conclusions: the two-timestamp model scores 87.85 PDMS / 30.68 EPDMS and LP-PAIR 88.44 / 30.56, both above every compared method on NavHard (best baseline: DVGT-2, 28.80), and the fusion comparisons use the same configuration in every arm. We will report both rankings in the paper.

Table R9. History length.

Geometry timestampsValPDMSEPDMS
178.0083.3028.68
281.3187.8530.68
3 (main)76.8288.2731.78
476.4388.2730.52

Table R10. Fusing features from different encoder layers.

SettingGeo. tapsVLMValPDMSEPDMS
Final-layer fusion (main)O23Final76.8288.2731.78
CAT3O4/O17/O23Final ×379.2588.3131.34
DUP3O23 ×3Final ×377.3988.1431.64
LP-LATEO17/O2328/2879.0088.0928.06
LP-PAIRO17/O2314/2879.8188.4430.56

Q4. Mapping Qwen3-VL tokens onto the grid; grid sensitivity. Each 1920×1080 view is encoded at native resolution, which the processor rounds to 1920×1088; with 16-pixel patches and 2×2 merging this gives a 34×60 grid of merged tokens. We take the final hidden states at the image-token positions, restore their 2D layout from the processor's grid size, and average-pool each 2×2 block, so each cell of the 17×30 grid summarizes a fixed 64×64-pixel region. VGGT-Ω features of the same view are pooled to the same grid, which defines corresponding row-major addresses. On sensitivity, an earlier front-camera geometry-only study gave 86.74 PDMS with a coarser 8×11 grid versus 87.37 with 17×30, and shifting the VLM grid by one cell against the geometry grid costs 2.53 PDMS / 4.14 EPDMS.

Q5. Closed-loop / reactive evaluation. See Table R1 in the general response: zero-shot closed loop on 420 HUGSIM scenarios with reactive vehicles. VLG-Drive reaches 25.9 HD-Score [22.9, 28.9], versus 21.2 for LTF, 16.1 for DVGT-2, and 10.7 for VGGDrive, with the highest NC, DAC, TTC, and route completion. Interactive evaluation on nuPlan remains future work.

Novelty. We agree that "new direction" is too broad a phrase and will replace it with a precise statement (see "Main idea" in the general response). VLG-Drive introduces a fusion paradigm that differs from prior semantic–geometric planners: instead of separate experts, geometry injected into a VLM, or geometry alone, it binds frozen VLM and geometry features cell by cell into joint tokens for a dedicated planner. The results above show that this paradigm is simple, effective, and stable.

Thanks for your reponse. Most of my concerns got adressed. Nevertheless it got uncler to me me what your validation sample size is. In your comment above you report 2,876 samples, while in C.2 you report 3,373 validation samples. Please clarify this. A few more things to consider:

  • Please clarify that "lightweight" refers only to training, because your inference is heavy with 1.12 s per step. Please report this in the paper.
  • Contrast your method better to VLG-Drive with Spatial-MLLM or VG LLM (the main idea section does only highlight the difference to UniDriveVLA, VGGDrive and DVGT-2)
  • Ablation arms are still single runs? For all experiments consider using at least three seeds.

Follow-up to Reviewer e2Em

We thank the reviewer for the follow-up and are glad that most concerns are resolved.

Validation sample size. Both numbers come from the same 40 held-out NavTrain logs, on which no model trains, and the 2,876 PDMS scenarios are a subset of the 3,373 samples. The 3,373 samples form the validation set we use during training, where we compute the open-loop trajectory error. The PDMS evaluation runs the full NAVSIM agent, which loads the camera images of every input timestamp. For 497 of the 3,373 samples, the NavTrain release does not include the camera images at one of the two earlier timestamps that the main model uses (−0.5 s or −1.0 s), because it ships images for only a subset of frames. During training, our data loader replaces such a frame with the nearest available one, so these samples remain in the training-time validation set; the PDMS split leaves them out. We will state both counts and how they relate in the paper.

"Lightweight" refers to training. We agree and will make this explicit. What is lightweight is the trained part: the fusion interface and planner, 106.1M parameters trained in about 58 GPU-h on cached features. At inference, VLG-Drive runs both frozen foundation models and needs 1.12 s and 9.8 GiB per planning step on one RTX 6000 Ada (Table R5), compared with 3.03 s for VGGDrive and 0.425 s for DVGT-2; the 2B VLM accounts for 65 of the 87 TFLOPs. We will restrict "lightweight" to training and report this latency in the main text.

Contrast with Spatial-MLLM and VG LLM. Both add 3D geometry to a multimodal LLM for spatial question answering. Spatial-MLLM merges the 2D visual features of Qwen2.5-VL with those of a VGGT-initialized spatial encoder through an MLP connector, and VG LLM adds VGGT features to the visual tokens of Qwen2.5-VL. In both, the fused tokens are the input of the language model, which is fine-tuned with instruction-style supervision and answers in text. VLG-Drive differs in three respects:

  • Where the fusion happens. We fuse at the output of the frozen VLM. The VLM features are computed from the images and the driving prompt alone, and the geometry features never pass through the language model.
  • Which feature anchors the joint token. Our joint token is anchored on the geometry feature, and the VLM feature of the same cell is injected into it. In our fusion study, anchoring on geometry outperforms anchoring on the VLM feature (Table R2: 88.06 / 30.75 vs. 87.78 / 29.47), the anchoring that these models use, although they fuse at the LLM input.
  • What reads the joint tokens. A dedicated planner, trained on trajectory labels, regresses continuous ego motion. Neither foundation model is fine-tuned, and planning involves no language decoding.

We will add this contrast to the statement of the main idea in the paper.

Seeds for the ablation arms. In the submission, the ablation arms were single runs. With our limited GPU resources, we could not repeat every study within the discussion period, so we prioritized the fusion study, the central ablation of the paper, and have now completed multiple runs for it: the independent-streams and geometry-only arms were each trained twice more, so that all three arms have three training runs (Table R11). On both benchmarks, every fusion run scores above every independent-streams run, and every independent-streams run above every geometry-only run. We will report these runs in the paper.

Table R11. Three training runs per arm of the fusion study (main recipe; PDMS / EPDMS).

ArmRun 1 (reported)Run 2Run 3Mean ± s.d.
Pointwise fusion (main)88.27 / 31.7888.35 / 31.8188.21 / 31.3988.28 ± 0.07 / 31.66 ± 0.23
Independent streams87.27 / 28.4387.54 / 30.4587.56 / 30.0187.46 ± 0.16 / 29.63 ± 1.06
Geometry only86.97 / 27.9986.76 / 27.1086.90 / 27.6286.88 ± 0.11 / 27.57 ± 0.45
Reviewer_mbP3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

VLG-Drive proposes a camera-only autonomous driving planner that combines features from frozen vision-language and geometric foundation models using a lightweight fusion interface. This main contribution is this interface, which is formulated as a geometry-anchored pointwise residual fusion. The fusion is carried out at corresponding camera-grid locations and replaces the token from two independent streams with joint tokens. The planned trajectory is predicted using an action conditioned Transformer decoder. The paper reports improved driving metrics compared to competing baselines.

Critical strengths
  • Well-motivated and useful contribution - The paper explores a meaningful question about combining the strengths of features from foundation models that excel at complementary tasks such as VLMs and geometry focused models. The paper also explores a suitable fusion mechanism (geometry-anchored pointwise residual fusion) that is lightweight to train.
  • Well-designed ablation studies - The ablation studies clearly demonstrate the benefit of fusing the two streams compared to using independent streams. The paper also provides ablation experiments to demonstrate the impact of the proposed fusion interface in comparison with possible alternative designs. The empirical results are generally strong (especially the NavHard results), conditional on a few clarifications (see questions).
Critical weaknesses
  • While it is clear that fusion of the two information streams is important from this work as well as VGGDrive, the main differentiating factor with respect to VGGDrive is when and how to fuse them. The benefits of the proposed method compared to VGGDrive are not clearly presented. A trajectory decoder is used in this paper that was not in VGGDrive - the impact of this choice is unclear. Overall, the training and inference cost/latency is not reported - this makes it hard to see how the proposed fusion method and the additional trajectory module fare in comparison with VGGDrive.
  • The analysis about when vision-language context and geometry contexts help can be useful. But the current discussion in section 4.6 in unsubstantiated. The claim should be clearly validated through additional models as well, for example, the ablation models from Table 3.
Questions for authors
  • In Table 3, the geometry carrier fusion is trained with the main recipe which uses 1024-width decoder whereas the other ablation baselines use 512-width decoder and limited training setup. This comparison seems unfair - can this be addressed by reporting the geometry carrier fusion result with the base training recipe as well?
  • Can the authors provide a comparison of training/inference cost and latency for the proposed method in comparison to the baselines?
  • One aspect that is not sufficiently ablated is the mechanism used to establish correspondence between the two streams. What happens if the calibration is not perfect? Will the experiment results hold if the correspondence is perturbed?
  • How much of the fusion improvement remains when the motion-history input is removed?
2 comments

Response to Reviewer mbP3 (1/2)

We thank the reviewer for the specific, actionable questions. Closed-loop results (Table R1) are in our general response.

Differentiation from VGGDrive. VGGDrive injects geometry into a VLM, fine-tunes it (8.48B trainable parameters), and generates the trajectory as text tokens. VLG-Drive keeps both foundation models frozen, fuses the two features at each shared camera-grid cell before readout, and decodes trajectories with a dedicated 106.1M-parameter planner. Under our evaluation of VGGDrive's official checkpoint, VLG-Drive is ahead on NavHard (31.78 vs. 26.09 EPDMS), NavTest (88.27 vs. 84.02 PDMS), and closed loop (25.9 vs. 10.7 HD-Score, Table R1), and runs 2.7× faster (Table R5). Because the two systems differ in both the fusion and the decoder, we do not attribute this gap to fusion alone. Our fusion claims rest on the matched studies (Tables R2 and R7), which hold the decoder fixed, and we have not isolated the decoder's own contribution.

Q1. Geometry-carrier fusion with the base recipe. We trained this row (Table R2): 88.06 PDMS / 30.75 EPDMS, versus 87.12 / 28.69 for independent streams and 86.87 / 29.18 for the no-injection geometry carrier. With every row on the base recipe, the geometry carrier is better than the VLM carrier both with fusion (88.06 vs. 87.78; 30.75 vs. 29.47) and without it (86.87 vs. 85.23; 29.18 vs. 25.89).

Table R2. Fusion study, all rows on the base recipe (decoder width 512, 12,000 updates, same optimizer, schedule, and data). The geometry-carrier fusion row and the NavHard column are new.

Dense representationInteractionPDMSEPDMS
Independent streamsDecoder attention only87.1228.69
VLM carrierNo dense geometry injection85.2325.89
VLM carrierPointwise fusion87.7829.47
Geometry carrierNo dense VLM injection86.8729.18
Geometry carrierPointwise fusion88.0630.75

Q2. Training/inference cost and latency. Training is cheap because only the interface and planner are trained on cached encoder outputs (about 58 GPU-h including the one-time caches). Inference is dominated by the 2B VLM (65 of 87 TFLOPs); VLG-Drive is 2.7× faster than VGGDrive, which generates about 99 tokens per step with a 7B model. Reaching real-time rates would require distilling the VLM or reusing its features across steps.

Table R5. Cost (one RTX 6000 Ada; HUGSIM input, three 1920×1080 cameras; FLOPs per planning step; †: reported by the authors).

MethodTotal paramsTrainableTraining (GPU-h)Latency (ms)Memory (GiB)TFLOPs
VLG-Drive3.38B106.1M29 (+29 once for caches)11239.887.0
Geometry only1.25B103.0M274505.321.7
VGGDrive9.39B8.48Bn/r302821.677.0
DVGT-21.71B1.71B≈15,000†42511.415.5

Response to Reviewer mbP3 (2/2)

Q3. Imperfect calibration / perturbed correspondence. The VLM–geometry pairing does not use calibration, since both grids index the same image; calibration enters only through the ray encoding. Table R6 perturbs it at test time. Small errors cost about one point or less, while a 3° rotation costs 5.26 PDMS / 4.94 EPDMS. Shifting the VLM grid by one cell with exact calibration costs 2.53 / 4.14, so the planner relies on the cell-wise correspondence and not only on the presence of both sources.

Table R6. Perturbed calibration and correspondence at test time (main model, no retraining; drawn per scenario and camera, applied to all input frames).

PerturbationPDMSEPDMS
None88.2731.78
Rotation 1°87.4930.99
Rotation 3°83.0126.84
Translation 10 cm87.2530.42
Focal length ±5%87.8731.88
VLM grid shifted by one cell85.7427.64

Q4. Fusion without the motion-history input. We removed the history string from the prompt, both when caching the training features and at test time, and retrained both arms with the main recipe (Table R7). Fusion is still ahead of independent streams by 0.19 PDMS and 2.35 EPDMS, and its 31.84 EPDMS is above every NavHard baseline. The geometry branch still sees three timestamps; with a single geometry timestamp the main model drops to 83.30 PDMS / 28.68 EPDMS.

Table R7. Without the motion-history prompt.

ArmPDMSEPDMS
Independent streams (I)87.6629.49
Pointwise fusion (F)87.8531.84

Validating the analysis of where VLM context helps with another model. We repeated the component analysis with DVGT-2 as the geometry backbone (Table R8). The pattern is the same as with VGGT-Ω: the gain concentrates in Stage 2 (+1.53 vs. +0.62 stage score), drivable-area compliance improves in both stages, and no-at-fault collisions are unchanged.

Table R8. NavHard stage scores and selected components, geometry only (G) vs. pointwise fusion (F).

BackboneStageNC (G → F)DAC (G → F)Stage score (G → F)
VGGT-ΩS197.78 → 97.3384.44 → 85.5674.06 → 74.87
VGGT-ΩS282.42 → 82.4368.56 → 71.7438.74 → 40.99
DVGT-2S196.56 → 95.7885.11 → 86.8972.76 → 73.38
DVGT-2S273.78 → 73.7562.73 → 63.6230.10 → 31.63
Reviewer_RjpgOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes VLG-DRIVE, a camera-based driving planner that combines frozen vision-language and geometric foundation-model features. It pairs features at corresponding camera-grid locations, injects VLM features into geometry features through a learned residual, and uses a dedicated transformer planner to regress future trajectories. Only the fusion interface and planner are trained. The method reaches 31.78 EPDMS on NavHard and 88.27 PDMS on NavTest.

Critical strengths

Clear, practical design: The method exploits the known spatial correspondence between the two feature streams, rather than leaving the planner to discover it from trajectory supervision.

  • Useful ablations: The paper studies independent streams, alternative fusion configurations, reduced training-label fractions, and a second geometry backbone. Fusion outperforms independent streams at all tested label fractions and improves both tested geometry backbones
  • Relevant component analysis: The reported NavHard results suggest that VLM context particularly helps drivable-area compliance and lane keeping after perturbation, while collision avoidance changes little.
Critical weaknesses

The novelty is meaningful but incremental. The main contribution is a specific fusion interface—spatially matched, geometry-anchored residual fusion—rather than a new foundation model or planning paradigm. The paper could better distinguish this contribution from prior feature-fusion and adapter-based approaches.

  • Some key fusion comparisons are not fully controlled. In Table 3, the main geometry-carrier fusion model uses the main recipe, while most other rows use the base recipe. This makes the headline comparison between pointwise fusion and independent streams less clean. The matched-recipe label-fraction experiments help support the fusion claim, but they do not resolve every concern about training-budget effects.
  • Uncertainty is not characterized. The paper reports point estimates, but no variation across random seeds or confidence intervals. This matters especially for smaller score differences.
  • The broad performance comparisons are not fully apples-to-apples. Baselines differ in camera count, sensor inputs, auxiliary supervision, and training scale. The results are promising, but they do not isolate the method’s advantage over those systems under matched conditions.
  • The evaluation scope is limited. The experiments use offline, non-reactive benchmarks; closed-loop performance and transfer beyond these benchmarks remain untested. The authors acknowledge this limitation.
Questions for authors
  1. Could you provide a fully matched comparison between pointwise fusion and independent streams? In Table 3, the main geometry-carrier fusion model uses the main training recipe, while most other configurations use the base recipe. Would the fusion advantage persist if all variants used the same decoder width, training steps, and optimization settings?

  2. How robust are the reported gains across random seeds? Please report results across multiple runs, or provide confidence intervals, particularly for comparisons with relatively small score differences.

  3. Can you evaluate VLG-DRIVE in a closed-loop setting? The current results are from offline, non-reactive benchmarks. Such an evaluation would help establish whether the reported planning gains carry over when the planner’s actions affect subsequent states.

  4. How should readers interpret comparisons with baselines using different inputs and training resources? For example, could you clarify which conclusions follow from matched comparisons, and which are system-level comparisons across different camera counts, sensors, or training regimes?

1 comment

We thank the reviewer for the detailed assessment. Closed-loop results (Table R1) are in our general response.

Q1. Fully matched pointwise fusion vs. independent streams. We agree the submitted fusion table mixed recipes. We completed it so that all rows use the same base recipe (Table R2): fusion reaches 88.06 PDMS / 30.75 EPDMS versus 87.12 / 28.69 for independent streams, with higher NC, DAC, TTC, and EP. The no-injection geometry carrier has the same memory layout as fusion and scores below independent streams on NavTest, so the gain is not from memory compaction.

Table R2. Fusion study, all rows on the base recipe (decoder width 512, 12,000 updates, same optimizer, schedule, and data). The geometry-carrier fusion row and the NavHard column are new.

Dense representationInteractionPDMSEPDMS
Independent streamsDecoder attention only87.1228.69
VLM carrierNo dense geometry injection85.2325.89
VLM carrierPointwise fusion87.7829.47
Geometry carrierNo dense VLM injection86.8729.18
Geometry carrierPointwise fusion88.0630.75

Table R3 gives a second fully matched comparison under the main recipe (width 1024, 24,000 updates): fusion is ahead of independent streams at 10%, 25%, and 100% of the labels on both benchmarks. The same holds without the motion-history prompt (F 87.85 / 31.84 vs. I 87.66 / 29.49).

Table R3. Reduced labels, main recipe (nested whole-log subsets of NavTrain; PDMS / EPDMS).

Representation10%25%100%
Geometry only (G)80.21 / 21.5484.33 / 24.0086.97 / 27.99
Independent streams (I)79.21 / 18.9684.62 / 23.5387.27 / 28.43
Pointwise fusion (F)82.23 / 21.5185.43 / 24.0388.27 / 31.78

Q2. Robustness across seeds. Table R4 reports three training runs of the main model; every run is above every NavHard baseline (best: DVGT-2, 28.80). For closed loop, Table R1 gives 95% scenario-bootstrap intervals, and rerunning VLG-Drive on 54 scenarios moves its mean HD-Score by only −0.4 points [−4.9, 4.1].

Table R4. Training runs of the main model.

RunPDMSEPDMS
Seed 0 (reported)88.2731.78
Seed 188.3531.81
Seed 0, second run88.2131.39
Mean ± s.d.88.28 ± 0.0731.66 ± 0.23

Q3. Closed-loop evaluation. See Table R1 in the general response. VLG-Drive reaches 25.9 HD-Score, ahead of LTF (+4.7 [1.1, 8.4]), VGGDrive (+14.6), and DVGT-2 (+11.1). Because the scenes come from nuScenes, Waymo, KITTI-360, and PandaSet, this also tests transfer beyond NAVSIM.

Q4. Matched vs. system-level conclusions. We will state this explicitly in the paper. Matched (same data, recipe, and decoder; only the studied factor changes): the fusion study, reduced labels, the no-history control, the second geometry backbone, feature levels, history length, and calibration. These support our claims about the fusion interface. System-level (baselines differ in cameras, LiDAR, auxiliary supervision, and training scale): the NavHard and NavTest comparisons with published methods and the closed-loop study. The NavTest comparison will list each method's camera count, LiDAR use, and auxiliary supervision. At system level we claim only the highest NavHard EPDMS among the compared methods and the highest closed-loop HD-Score among the compared planners; we do not claim the best NavTest score, where DVGT-2 (88.60) and DVGT-2-NAVSIM (90.30) are higher. VGGDrive and DVGT-2 are evaluated by us from their official checkpoints with documented inputs.

Novelty. The novelty is the fusion paradigm itself (see "Main idea" in the general response). Prior semantic–geometric planners decouple the two sources into separate experts (UniDriveVLA), inject geometry into a VLM that generates the plan (VGGDrive), or drop semantics (DVGT-2). VLG-Drive binds frozen VLM and geometry features cell by cell into joint tokens for a dedicated planner. The matched studies show that this binding, not access to both sources, drives the gain (Tables R2 and R3), and that it holds with a second geometry backbone.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

We thank the reviewers for their careful reading and constructive questions. We ran the requested experiments and will add them to the paper; each is reported in the response to the reviewer who asked for it. NavTest PDMS and NavHard EPDMS are in percent.

Main idea. Existing planners combine vision-language semantics and 3D geometry in different ways: UniDriveVLA routes them to separate experts, VGGDrive injects geometry into a VLM that generates the plan as text, and DVGT-2 plans from geometry alone. VLG-Drive uses a different fusion paradigm. It fuses the VLM feature and the geometry feature that describe the same camera-grid cell into one joint token, anchored on geometry, and a dedicated planner reads these tokens; both foundation models stay frozen. This paradigm is

  • simple: a shared two-layer residual MLP applied at each cell on top of frozen encoders, with 106.1M trainable parameters and about 58 GPU-h of training including feature caching;
  • effective: the highest NavHard EPDMS among the compared methods (31.78) and the highest zero-shot closed-loop HD-Score (Table R1 below), at 2.7× the speed of VGGDrive;
  • stable: fusion stays ahead of independent streams under both training recipes, at every label fraction, and without the motion-history prompt; the same interface improves a second geometry backbone (DVGT-2: +0.82 PDMS, +0.91 EPDMS over geometry only); and small calibration errors cost about one point or less.

The controlled studies also show that the gain comes from the cell-wise binding itself: the same memory layout without VLM injection scores lower, and misaligning the two grids by one cell costs 4.14 EPDMS. Details are in the individual responses.

Zero-shot closed-loop driving (Rjpg Q3, e2Em Q5). We drive the NAVSIM-trained planners in closed loop on HUGSIM: 420 scenarios from nuScenes, Waymo, KITTI-360, and PandaSet with reactive inserted vehicles. No model sees HUGSIM data in training or tuning.

Table R1. Zero-shot closed loop on HUGSIM (brackets: 95% scenario-bootstrap interval).

MethodNCDACTTCCOMRcHD-Score
LTF43.085.238.893.435.721.2 [18.2, 24.3]
VGGDrive49.771.139.897.323.210.7 [9.1, 12.5]
DVGT-230.182.527.099.433.616.1 [13.2, 19.1]
VLG-Drive59.289.251.698.240.625.9 [22.9, 28.9]

Paired HD-Score differences: +4.7 over LTF [1.1, 8.4]; +14.6 over VGGDrive [11.1, 18.2] and +11.1 over DVGT-2 [6.9, 15.2] on the 315 scenarios with paired episodes. VLG-Drive has the highest collision, drivable-area, time-to-collision, and route-completion scores.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.