- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces VLG-Drive, a driving planner built on two frozen foundation models (2B Qwen3-VL and VGGT-Ω). Features from both are pooled onto a shared grid per camera across three cameras and tagged with Plücker encodings. A zero-initialized residual MLP then fuses each VLM–geometry pair into a single geometry-anchored token, which replaces both dense streams in the planner memory. A command-conditioned query decoder regresses eight incremental ego poses. Training is restricted to the interface and planner. The method reaches 31.78 EPDMS on NavHard, the best among compared methods, and 88.27 PDMS on NavTest with cameras only. Ablation studies examine: fusion vs. independent streams vs. no-injection carriers, reduced label fractions, a second geometry backbone (DVGT-2), feature-level selection, geometry history length, attention and intervention analyses.
Critical strengths
- The interface is minimal and well motivated. Exploiting the known pixel-level correspondence instead of letting decoder attention discover it is a clean idea, and zero-initialization from the geometry anchor is a sensible design choice.
- The no-injection carrier separates fusion from memory compaction. The label-fraction study is matched to the main recipe.
- The analysis goes beyond attention maps. Mean-feature interventions on top-attended and path-tube cells, with random controls, give causal evidence of where the planner reads.
- The paper is very clearly written, and the implementation and protocol details are thorough.
Critical weaknesses
- Confounded fusion study (Table 3). The headline row, geometry carrier with pointwise fusion, uses the main recipe. All other rows use the base recipe. This confounds both the fusion vs. independent streams comparison and the carrier comparison. A base-recipe geometry-carrier fusion row is needed.
- No variance estimates. All results are single runs.
- Hyperparameter selection on test benchmarks. History length (Table 8) and feature levels (Table 6) appear to be chosen from NavTest/NavHard scores. The reported best configuration is therefore partially selected on the evaluation sets. A held-out validation split should drive these choices.
- Unclear driving supervision. Table 2 lists “None” for auxiliary supervision, yet the VLM is “driving-adapted”. The paper should state which model and data were used for this adaptation. In particular, it should say whether that data overlaps with nuPlan logs underlying NavTest/NavHard.
- Limited novelty. Adding aligned geometry features to VLM visual tokens through an MLP or residual connector is already used. The main difference is decoding with a dedicated planner rather than the LLM. Claiming a new direction overstates this. The contribution is a well-executed combination rather than a new mechanism.
- Missing computational cost analysis. Each planning step runs a 2B VLM plus VGGT-Ω over three cameras × three frames, but no latency, FLOPs, or memory figures are reported. Training on cached features hides this cost. Without these numbers, the practical appeal of “lightweight” is unclear.
Questions for authors
- What is the driving-adapted Qwen3-VL? Is there any overlap with NAVSIM train or test logs?
- Can you report geometry-carrier fusion under the base recipe, and seed variance for the main NavHard/NavTest numbers and the key ablations?
- What are the end-to-end inference latency and GPU memory compared with DVGT-2 and DiffusionDrive?
- How are Qwen3-VL image tokens, with dynamic resolution and patch merging, mapped onto the grid? How sensitive are results to grid resolution?
- Have you run any closed-loop or reactive evaluation?
4 comments
Response to Reviewer e2Em (2/2)
History length and feature levels. These two studies are not how the main configuration was tuned. The main model, final-layer fusion with three geometry timestamps, is the reference configuration on which every other experiment is built (the fusion study, reduced labels, the second backbone, the no-history control, calibration, and closed loop). The history-length and feature-level studies are follow-up analyses of this design: they ask how much visual history the geometry branch needs and whether fusing intermediate features adds information beyond the final layer.
NAVSIM provides no official validation split, and results are commonly reported on NavTest directly. To address the concern nonetheless, we evaluated both studies on a held-out split of 2,876 scenarios from NavTrain logs that no model trains on, scored with the NavTest PDMS protocol at the final checkpoint (Val in Tables R9 and R10). It agrees with the test benchmarks on the main findings: a second timestamp gives the largest improvement, and depth-paired multi-level fusion (LP-PAIR) improves on final-layer fusion, with the best held-out and NavTest PDMS. It differs in preferring two timestamps over three. Neither difference affects our conclusions: the two-timestamp model scores 87.85 PDMS / 30.68 EPDMS and LP-PAIR 88.44 / 30.56, both above every compared method on NavHard (best baseline: DVGT-2, 28.80), and the fusion comparisons use the same configuration in every arm. We will report both rankings in the paper.
Table R9. History length.
| Geometry timestamps | Val | PDMS | EPDMS |
|---|---|---|---|
| 1 | 78.00 | 83.30 | 28.68 |
| 2 | 81.31 | 87.85 | 30.68 |
| 3 (main) | 76.82 | 88.27 | 31.78 |
| 4 | 76.43 | 88.27 | 30.52 |
Table R10. Fusing features from different encoder layers.
| Setting | Geo. taps | VLM | Val | PDMS | EPDMS |
|---|---|---|---|---|---|
| Final-layer fusion (main) | O23 | Final | 76.82 | 88.27 | 31.78 |
| CAT3 | O4/O17/O23 | Final ×3 | 79.25 | 88.31 | 31.34 |
| DUP3 | O23 ×3 | Final ×3 | 77.39 | 88.14 | 31.64 |
| LP-LATE | O17/O23 | 28/28 | 79.00 | 88.09 | 28.06 |
| LP-PAIR | O17/O23 | 14/28 | 79.81 | 88.44 | 30.56 |
Q4. Mapping Qwen3-VL tokens onto the grid; grid sensitivity. Each 1920×1080 view is encoded at native resolution, which the processor rounds to 1920×1088; with 16-pixel patches and 2×2 merging this gives a 34×60 grid of merged tokens. We take the final hidden states at the image-token positions, restore their 2D layout from the processor's grid size, and average-pool each 2×2 block, so each cell of the 17×30 grid summarizes a fixed 64×64-pixel region. VGGT-Ω features of the same view are pooled to the same grid, which defines corresponding row-major addresses. On sensitivity, an earlier front-camera geometry-only study gave 86.74 PDMS with a coarser 8×11 grid versus 87.37 with 17×30, and shifting the VLM grid by one cell against the geometry grid costs 2.53 PDMS / 4.14 EPDMS.
Q5. Closed-loop / reactive evaluation. See Table R1 in the general response: zero-shot closed loop on 420 HUGSIM scenarios with reactive vehicles. VLG-Drive reaches 25.9 HD-Score [22.9, 28.9], versus 21.2 for LTF, 16.1 for DVGT-2, and 10.7 for VGGDrive, with the highest NC, DAC, TTC, and route completion. Interactive evaluation on nuPlan remains future work.
Novelty. We agree that "new direction" is too broad a phrase and will replace it with a precise statement (see "Main idea" in the general response). VLG-Drive introduces a fusion paradigm that differs from prior semantic–geometric planners: instead of separate experts, geometry injected into a VLM, or geometry alone, it binds frozen VLM and geometry features cell by cell into joint tokens for a dedicated planner. The results above show that this paradigm is simple, effective, and stable.
Thanks for your reponse. Most of my concerns got adressed. Nevertheless it got uncler to me me what your validation sample size is. In your comment above you report 2,876 samples, while in C.2 you report 3,373 validation samples. Please clarify this. A few more things to consider:
- Please clarify that "lightweight" refers only to training, because your inference is heavy with 1.12 s per step. Please report this in the paper.
- Contrast your method better to VLG-Drive with Spatial-MLLM or VG LLM (the main idea section does only highlight the difference to UniDriveVLA, VGGDrive and DVGT-2)
- Ablation arms are still single runs? For all experiments consider using at least three seeds.
Follow-up to Reviewer e2Em
We thank the reviewer for the follow-up and are glad that most concerns are resolved.
Validation sample size. Both numbers come from the same 40 held-out NavTrain logs, on which no model trains, and the 2,876 PDMS scenarios are a subset of the 3,373 samples. The 3,373 samples form the validation set we use during training, where we compute the open-loop trajectory error. The PDMS evaluation runs the full NAVSIM agent, which loads the camera images of every input timestamp. For 497 of the 3,373 samples, the NavTrain release does not include the camera images at one of the two earlier timestamps that the main model uses (−0.5 s or −1.0 s), because it ships images for only a subset of frames. During training, our data loader replaces such a frame with the nearest available one, so these samples remain in the training-time validation set; the PDMS split leaves them out. We will state both counts and how they relate in the paper.
"Lightweight" refers to training. We agree and will make this explicit. What is lightweight is the trained part: the fusion interface and planner, 106.1M parameters trained in about 58 GPU-h on cached features. At inference, VLG-Drive runs both frozen foundation models and needs 1.12 s and 9.8 GiB per planning step on one RTX 6000 Ada (Table R5), compared with 3.03 s for VGGDrive and 0.425 s for DVGT-2; the 2B VLM accounts for 65 of the 87 TFLOPs. We will restrict "lightweight" to training and report this latency in the main text.
Contrast with Spatial-MLLM and VG LLM. Both add 3D geometry to a multimodal LLM for spatial question answering. Spatial-MLLM merges the 2D visual features of Qwen2.5-VL with those of a VGGT-initialized spatial encoder through an MLP connector, and VG LLM adds VGGT features to the visual tokens of Qwen2.5-VL. In both, the fused tokens are the input of the language model, which is fine-tuned with instruction-style supervision and answers in text. VLG-Drive differs in three respects:
- Where the fusion happens. We fuse at the output of the frozen VLM. The VLM features are computed from the images and the driving prompt alone, and the geometry features never pass through the language model.
- Which feature anchors the joint token. Our joint token is anchored on the geometry feature, and the VLM feature of the same cell is injected into it. In our fusion study, anchoring on geometry outperforms anchoring on the VLM feature (Table R2: 88.06 / 30.75 vs. 87.78 / 29.47), the anchoring that these models use, although they fuse at the LLM input.
- What reads the joint tokens. A dedicated planner, trained on trajectory labels, regresses continuous ego motion. Neither foundation model is fine-tuned, and planning involves no language decoding.
We will add this contrast to the statement of the main idea in the paper.
Seeds for the ablation arms. In the submission, the ablation arms were single runs. With our limited GPU resources, we could not repeat every study within the discussion period, so we prioritized the fusion study, the central ablation of the paper, and have now completed multiple runs for it: the independent-streams and geometry-only arms were each trained twice more, so that all three arms have three training runs (Table R11). On both benchmarks, every fusion run scores above every independent-streams run, and every independent-streams run above every geometry-only run. We will report these runs in the paper.
Table R11. Three training runs per arm of the fusion study (main recipe; PDMS / EPDMS).
| Arm | Run 1 (reported) | Run 2 | Run 3 | Mean ± s.d. |
|---|---|---|---|---|
| Pointwise fusion (main) | 88.27 / 31.78 | 88.35 / 31.81 | 88.21 / 31.39 | 88.28 ± 0.07 / 31.66 ± 0.23 |
| Independent streams | 87.27 / 28.43 | 87.54 / 30.45 | 87.56 / 30.01 | 87.46 ± 0.16 / 29.63 ± 1.06 |
| Geometry only | 86.97 / 27.99 | 86.76 / 27.10 | 86.90 / 27.62 | 86.88 ± 0.11 / 27.57 ± 0.45 |
Response to Reviewer e2Em (1/2)
We thank the reviewer for the thorough review. Closed-loop results (Table R1) are in our general response.
Q1. The driving-adapted Qwen3-VL and data overlap. It is the publicly released Stage-1 nuScenes checkpoint of UniDriveVLA (2B), fine-tuned from Qwen3-VL-2B-Instruct. According to its released training configuration, its data are driving QA and planning corpora built on nuScenes and on BDD-X, DRAMA, LingoQA, CODA-LM, Talk2Car, SUTD-TrafficQA, and MAPLM, plus a general VQA subset. None is derived from nuPlan, the source of the NAVSIM logs, so no NAVSIM training or test log reaches the VLM during its adaptation. We will state this in the paper.
Q2a. Geometry-carrier fusion under the base recipe. We trained this row (Table R2): 88.06 PDMS / 30.75 EPDMS, versus 87.12 / 28.69 for independent streams and 86.87 / 29.18 for the no-injection geometry carrier, which has the same memory layout as fusion. Under the main recipe, fusion is also ahead of independent streams at 10%, 25%, and 100% of the labels on both benchmarks.
Table R2. Fusion study, all rows on the base recipe (decoder width 512, 12,000 updates, same optimizer, schedule, and data). The geometry-carrier fusion row and the NavHard column are new.
Q2b. Seed variance. Table R4: three training runs of the main model; every run is above every NavHard baseline. Closed-loop results include 95% bootstrap intervals (Table R1).
Table R4. Training runs of the main model.
Q3. Latency, memory, FLOPs. VLG-Drive takes 1123 ms and 9.8 GiB per step against 425 ms and 11.4 GiB for DVGT-2 (Table R5). The 2B VLM dominates our cost (65 of 87 TFLOPs); the geometry-only variant takes 450 ms. On the training side, VLG-Drive trains 106.1M parameters in about 58 GPU-h including the one-time caches, while DVGT-2 reports about 15,000 GPU-h.
Table R5. Cost (one RTX 6000 Ada; HUGSIM input, three 1920×1080 cameras; FLOPs per planning step; †: reported by the authors).