- Rating
- 3(+1)
- Confidence
- 1(−2)
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper presents PhysDrive, a vision-language model for driving understanding and planning that uses current and future 3D occupancy as training-time supervision for its internal representations. Built on Qwen2.5-VL, the method introduces dedicated grounding tokens supervised to capture current scene geometry and future scene evolution, with additional region-of-interest supervision for question-relevant grounding in understanding tasks. Training first combines supervised fine-tuning with these grounding objectives, then applies GRPO to improve trajectory generation using driving-oriented rewards while retaining the grounding supervision. Experiments on NuInstruct, nuScenes, and NAVSIM report improvements in driving understanding and planning, with ablations examining the contributions of the grounding signals and the number of grounding tokens.
Critical strengths
-
Well-motivated use of spatial and temporal supervision. PhysDrive combines current and future occupancy supervision to encourage representations that capture both scene geometry and its evolution. Table 4 supports the added value of future occupancy: compared with current-occupancy supervision alone, it reduces the collision rate from 0.29% to 0.22% and the intersection rate from 1.85% to 1.42%. These improvements provide concrete evidence that the temporal supervision benefits planning.
-
Benefits across driving understanding and planning. The proposed supervision improves distance estimation, state prediction, and risk grounding on NuInstruct, alongside planning improvements on nuScenes and NAVSIM. This demonstrates usefulness across multiple driving tasks. The subsequent RL stage further raises NAVSIM PDMS from 87.52 to 90.25, supporting the practical effectiveness of the overall training recipe.
Critical weaknesses
-
Limited evidence of generality across model scales and families. The method is evaluated only on Qwen2.5-VL at 3B and 7B, with the RL experiments and main ablations concentrated on the 3B model. This leaves it unclear whether the benefits of the proposed grounding supervision extend to larger models or different VLM architectures.
-
The contribution of grounding to the final RL results is not fully isolated. Table 4 supports the benefits of occupancy supervision during supervised training. However, the paper does not report a same-backbone baseline trained without grounding and subsequently optimized with the same RL recipe. It also does not compare retaining versus removing the grounding loss during RL from the same PhysDrive SFT checkpoint. These controls, with matched RL data, rewards, and training budgets, would clarify whether grounding provides additional benefits beyond RL alone and whether maintaining grounding supervision during RL is necessary for the reported performance.
Questions for authors
-
Do the improvements from current and future occupancy supervision extend to another VLM family or a larger backbone? A targeted baseline/current-only/current-plus-future comparison would help assess the generality of the method.
-
How does a same-backbone model trained without occupancy grounding perform after receiving the same GRPO training, with matched data, rewards, and training budgets? This would clarify whether grounding provides additional benefits after RL.
-
Starting from the same PhysDrive SFT checkpoint, how does retaining versus removing the grounding loss during RL affect planning performance and current/future occupancy prediction quality? This would directly test whether the retained supervision preserves the learned representations and benefits planning.
We appreciate the reviewer's insightful comments on the generalizability of PhysDrive and the interplay between physical grounding and RL. To address these questions, we conduct additional experiments and present the corresponding analyses below.
Q1. Generalization to Another VLM Family
To further evaluate the generalizability of PhysDrive, we extend our method to another VLM family, InternVL3-8B. All experiments are conducted under the SFT setting without RL. We compare three configurations on the nuScenes open-loop planning benchmark: the baseline model without physical grounding, the model with current occupancy supervision, and the model with full 4D physical grounding. The results are summarized below.
As shown, current occupancy supervision improves the average L2 error and intersection rate, while full 4D physical grounding further improves planning performance across all three average metrics. In particular, introducing 4D physical grounding reduces the average collision rate from 0.208% to 0.195% and the intersection rate from 2.181% to 1.237%, achieving a relative reduction of 43.3% in the latter. Together with our original results on Qwen2.5-VL-3B and Qwen2.5-VL-7B, these findings suggest that the benefits of 4D physical grounding extend across different VLM families and model scales.
Q2. Contribution of Physical Grounding to Subsequent RL
We conduct a controlled comparison to investigate whether physical grounding provides additional benefits after RL, using the same backbone, data, rewards, and training budgets. As shown in the table below, the Qwen2.5-VL-3B baseline achieves a PDMS of 84.03 after SFT, which further improves to 85.97 (+1.94) with RL. In contrast, incorporating our 4D physical grounding during SFT improves PDMS to 87.52. With the same RL training, PhysDrive further achieves 90.25 (+2.73), outperforming the baseline after RL by 4.28 points. Notably, PhysDrive also achieves a larger improvement during RL than the baseline (+2.73 vs. +1.94). These results demonstrate that the benefits of 4D physical grounding persist after RL, while the larger RL improvement further suggests that physically grounded representations provide a stronger foundation for subsequent policy optimization.
Q3. Effect of Retaining Grounding Supervision During RL
Our decision to retain physical grounding supervision during RL was driven by our experimentl results. We found that maintaining the grounding loss during policy optimization yielded slightly better planning performance.
As shown in the above table provided in our response to Q2, starting from the same PhysDrive SFT checkpoint, removing grounding supervision during RL yields a PDMS of 89.96, whereas retaining it achieves 90.25 (+0.29). We also observe improvements in no-at-fault collisions (98.84 → 98.96), drivable-area compliance (97.30 → 97.65), and ego progress (83.02 → 83.27). Although the overall gain is modest, these findings provide the empirical motivation for retaining grounding supervision in our original RL objective (Eq. 7).