acceptodds
Under review as a conference paper at ICLR 2027

PhysDrive: A Physically Grounded Vision-Language Model for Driving Understanding and Planning

Abstract

Vision-Language-Action (VLA) models have emerged as a promising paradigm for autonomous driving, supporting both scene understanding and motion planning. However, existing VLA models are typically optimized primarily with downstream outputs such as language responses or expert trajectories, providing limited supervision for the intermediate representations that connect visual perception to decision making. This output-centric training may encourage shortcut learning and hinder the model from capturing the underlying 3D geometry and physical dynamics of driving scenes. To address this limitation, we propose PhysDrive, a physically grounded vision-language model that uses 4D occupancy as structured training-time supervision for its internal representations. Specifically, PhysDrive decomposes 4D occupancy supervision into two complementary components: the current occupancy grounds the model in the 3D geometry of the observed scene, while future occupancy guides the modeling of its physical evolution over time. Together, these signals encourage PhysDrive to encode spatial structure and scene dynamics before producing downstream understanding or planning outputs. We further introduce a reinforcement learning framework to align generated trajectories with driving-oriented objectives, including safety, drivable-area compliance, and progress. Extensive experiments on NuInstruct, nuScenes, and NAVSIM demonstrate that PhysDrive consistently improves driving understanding and planning, demonstrating the effectiveness of physically grounded representations for downstream driving tasks.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Autonomous Driving

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_z9aKOpenReview ↗
Rating
3(+1)
Confidence
1(−2)
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents PhysDrive, a vision-language model for driving understanding and planning that uses current and future 3D occupancy as training-time supervision for its internal representations. Built on Qwen2.5-VL, the method introduces dedicated grounding tokens supervised to capture current scene geometry and future scene evolution, with additional region-of-interest supervision for question-relevant grounding in understanding tasks. Training first combines supervised fine-tuning with these grounding objectives, then applies GRPO to improve trajectory generation using driving-oriented rewards while retaining the grounding supervision. Experiments on NuInstruct, nuScenes, and NAVSIM report improvements in driving understanding and planning, with ablations examining the contributions of the grounding signals and the number of grounding tokens.

Critical strengths
  1. Well-motivated use of spatial and temporal supervision. PhysDrive combines current and future occupancy supervision to encourage representations that capture both scene geometry and its evolution. Table 4 supports the added value of future occupancy: compared with current-occupancy supervision alone, it reduces the collision rate from 0.29% to 0.22% and the intersection rate from 1.85% to 1.42%. These improvements provide concrete evidence that the temporal supervision benefits planning.

  2. Benefits across driving understanding and planning. The proposed supervision improves distance estimation, state prediction, and risk grounding on NuInstruct, alongside planning improvements on nuScenes and NAVSIM. This demonstrates usefulness across multiple driving tasks. The subsequent RL stage further raises NAVSIM PDMS from 87.52 to 90.25, supporting the practical effectiveness of the overall training recipe.

Critical weaknesses
  1. Limited evidence of generality across model scales and families. The method is evaluated only on Qwen2.5-VL at 3B and 7B, with the RL experiments and main ablations concentrated on the 3B model. This leaves it unclear whether the benefits of the proposed grounding supervision extend to larger models or different VLM architectures.

  2. The contribution of grounding to the final RL results is not fully isolated. Table 4 supports the benefits of occupancy supervision during supervised training. However, the paper does not report a same-backbone baseline trained without grounding and subsequently optimized with the same RL recipe. It also does not compare retaining versus removing the grounding loss during RL from the same PhysDrive SFT checkpoint. These controls, with matched RL data, rewards, and training budgets, would clarify whether grounding provides additional benefits beyond RL alone and whether maintaining grounding supervision during RL is necessary for the reported performance.

Questions for authors
  1. Do the improvements from current and future occupancy supervision extend to another VLM family or a larger backbone? A targeted baseline/current-only/current-plus-future comparison would help assess the generality of the method.

  2. How does a same-backbone model trained without occupancy grounding perform after receiving the same GRPO training, with matched data, rewards, and training budgets? This would clarify whether grounding provides additional benefits after RL.

  3. Starting from the same PhysDrive SFT checkpoint, how does retaining versus removing the grounding loss during RL affect planning performance and current/future occupancy prediction quality? This would directly test whether the retained supervision preserves the learned representations and benefits planning.

1 comment

We appreciate the reviewer's insightful comments on the generalizability of PhysDrive and the interplay between physical grounding and RL. To address these questions, we conduct additional experiments and present the corresponding analyses below.

Q1. Generalization to Another VLM Family

To further evaluate the generalizability of PhysDrive, we extend our method to another VLM family, InternVL3-8B. All experiments are conducted under the SFT setting without RL. We compare three configurations on the nuScenes open-loop planning benchmark: the baseline model without physical grounding, the model with current occupancy supervision, and the model with full 4D physical grounding. The results are summarized below.

MethodL2 1s ↓L2 2s ↓L2 3s ↓L2 Avg. ↓Coll. 1s ↓Coll. 2s ↓Coll. 3s ↓Coll. Avg. ↓Inter. 1s ↓Inter. 2s ↓Inter. 3s ↓Inter. Avg. ↓
InternVL3-8B0.1380.2950.5590.3310.0000.1170.5080.2080.2541.7394.5522.181
InternVL3-8B + Cur. Occ.0.1350.2840.5360.3180.0000.1760.6060.2600.2931.5434.3372.058
InternVL3-8B + 4D Occ.0.1360.2820.5190.3120.0000.1170.4690.1950.1950.9382.5791.237

As shown, current occupancy supervision improves the average L2 error and intersection rate, while full 4D physical grounding further improves planning performance across all three average metrics. In particular, introducing 4D physical grounding reduces the average collision rate from 0.208% to 0.195% and the intersection rate from 2.181% to 1.237%, achieving a relative reduction of 43.3% in the latter. Together with our original results on Qwen2.5-VL-3B and Qwen2.5-VL-7B, these findings suggest that the benefits of 4D physical grounding extend across different VLM families and model scales.

Q2. Contribution of Physical Grounding to Subsequent RL

We conduct a controlled comparison to investigate whether physical grounding provides additional benefits after RL, using the same backbone, data, rewards, and training budgets. As shown in the table below, the Qwen2.5-VL-3B baseline achieves a PDMS of 84.03 after SFT, which further improves to 85.97 (+1.94) with RL. In contrast, incorporating our 4D physical grounding during SFT improves PDMS to 87.52. With the same RL training, PhysDrive further achieves 90.25 (+2.73), outperforming the baseline after RL by 4.28 points. Notably, PhysDrive also achieves a larger improvement during RL than the baseline (+2.73 vs. +1.94). These results demonstrate that the benefits of 4D physical grounding persist after RL, while the larger RL improvement further suggests that physically grounded representations provide a stronger foundation for subsequent policy optimization.

MethodNCDACEPTTCComfortPDMS
Qwen2.5-VL-3B SFT97.9392.5277.9394.2099.9584.03
Qwen2.5-VL-3B SFT + RL98.3495.9376.8295.7499.9885.97
PhysDrive-3B SFT98.7095.2281.0995.8699.9987.52
PhysDrive-3B SFT + RL w/o Grounding98.8497.3083.0296.9599.9789.96
PhysDrive-3B SFT + RL w/ Grounding98.9697.6583.2796.9099.9890.25

Q3. Effect of Retaining Grounding Supervision During RL

Our decision to retain physical grounding supervision during RL was driven by our experimentl results. We found that maintaining the grounding loss during policy optimization yielded slightly better planning performance.

As shown in the above table provided in our response to Q2, starting from the same PhysDrive SFT checkpoint, removing grounding supervision during RL yields a PDMS of 89.96, whereas retaining it achieves 90.25 (+0.29). We also observe improvements in no-at-fault collisions (98.84 → 98.96), drivable-area compliance (97.30 → 97.65), and ego progress (83.02 → 83.27). Although the overall gain is modest, these findings provide the empirical motivation for retaining grounding supervision in our original RL objective (Eq. 7).

Reviewer_5LSPOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PhysDrive, a VLA model that uses current and future semantic occupancy as training supervision for driving understanding and trajectory planning. Dedicated grounding tokens interact with visual features and are supervised through depth and occupancy prediction, with additional region-of-interest supervision for question-relevant objects in understanding tasks. A subsequent GRPO stage optimizes driving rewards while retaining the grounding objective to preserve the learned representations. Experiments on NuInstruct, nuScenes, and NAVSIM show improvements in scene understanding and offline planning.

Critical strengths
  1. PhysDrive provides explicit supervision for the scene information that its internal representations are intended to capture. Current occupancy constrains 3D geometry, future occupancy supplies temporal supervision, and the optional region-of-interest objective connects understanding questions to relevant objects. This offers a concrete way to train spatial and temporal representations while retaining image inputs and a shared autoregressive output interface for language responses and trajectories.

  2. The supervision ablations provide evidence that future occupancy adds value beyond current-scene geometry. In Table 4, adding future occupancy to current occupancy reduces the planning collision rate from 0.29% to 0.22% and the intersection rate from 1.85% to 1.42%, while the full understanding model improves numerical MAE from 3.9 to 2.1 relative to the baseline. These comparisons support the usefulness of the proposed supervision across both planning and understanding, rather than relying only on comparisons between independently trained systems.

Critical weaknesses
  1. The experiments establish that occupancy supervision improves downstream performance, but do not fully establish what temporal information the model learns or how it uses that information to plan. Current and future occupancy are assessed through qualitative visualizations, without quantitative forecasting results that separate moving objects from largely static scene structure. The occupancy decoder also receives visual features, so successful reconstruction alone would not demonstrate that the dedicated grounding tokens supply the information responsible for better trajectories. A comparison with equally sized current-occupancy-only supervision, together with targeted interventions on the dynamic tokens, would help distinguish useful future modeling from the broader benefit of additional dense supervision.

  2. The planning evaluation does not assess the learned policy in interactive closed-loop driving in photorealistic environments. nuScenes measures predictions against recorded scenes, and the NAVSIM results do not establish performance when the model's own actions change subsequent observations and interactions. Consequently, lower offline collision and intersection rates do not demonstrate recovery from accumulated mistakes or improved safety around responding traffic. This is particularly relevant to the reinforcement-learning stage, since improving the training reward may not produce the same benefit when the policy repeatedly acts on states induced by its earlier decisions.

Questions for authors
  1. Authors are encouraged to conduct photorealistic closed-loop environments through HUGSIM(https://github.com/hyzhou404/HUGSIM) , WorldEngine(https://github.com/OpenDriveLab/WorldEngine) or NavSafe-∞ (https://github.com/VAIL-UCLA/NavSafe-infty). It's also better if the authors could accompany with visualizations/videos showing the planning execution traces.
  2. Do the generated trajectories depend on the scene information represented by the dynamic grounding tokens? Could you hold the images and instruction fixed, replace or perturb the dynamic tokens while preserving the token layout, and test whether trajectories change consistently with the altered scene prediction? A matched reinforcement-learning comparison with and without occupancy supervision would also clarify how much of the final planning performance depends on grounding.
3 comments

We thank the reviewer for the constructive feedback and valuable suggestions regarding closed-loop evaluation, the influence of dynamic latent tokens on trajectory planning, and the contribution of physical grounding to planning performance. We provide detailed responses to these points below.

Q1. Closed-Loop Evaluation on HUGSIM

We evaluate PhysDrive on HUGSIM-nuScenes and the results are shown as below.

MethodEasyMediumHardExtremeHD-Score
LTF86.2423.1024.0313.1234.17
DiffusionDrive50.3814.3013.7310.2920.74
PhysDrive SFT74.3645.1933.8810.7941.81
PhysDrive RL72.9847.2139.5010.4343.38

We compare our method with two representative planning baselines, LTF and DiffusionDrive. As shown in the table above, PhysDrive-RL achieves the highest overall HD-Score of 43.38, followed by PhysDrive-SFT with 41.81. Compared with LTF and DiffusionDrive, PhysDrive demonstrates substantial advantages in medium and hard scenarios. We add the visualizations of the closed-loop planning execution in Appendix A.3 (Figure 6) of the revised submission, illustrating how PhysDrive avoids collisions by adjusting its speed or heading, while LTF collides with preceding vehicles in the same scenarios. These results demonstrate the effectiveness of PhysDrive in challenging closed-loop driving scenarios and show that RL further improves planning performance, particularly under medium and hard driving conditions.

Q2. The Influence of the Dynamic Latent Tokens

Yes, the generated trajectories do depend on the scene information represented by the dynamic grounding tokens. Following your suggestion, we conduct a controlled intervention by replacing the dynamic latent tokens in the KV cache with those associated with different driving maneuvers. We add the visualization of this experiment in Appendix A.4 (Figure 8). As shown, with the input images and instructions held fixed, replacing the dynamic latent tokens leads to distinct planning trajectories. We additionally decode the corresponding future occupancy predictions, which exhibit changes in the predicted scene evolution. These observations suggest that dynamic latent tokens function as a form of latent action, encoding maneuver-related information that can be decoded into future scene evolution through the occupancy decoder or translated into planning waypoints through the autoregressive language decoder. This shared role helps explain why intervening on the dynamic latent tokens affects both future occupancy predictions and planning trajectories.

Q3. The Contribution of Physical Grounding to Planning Performance

We report the following experimental results to clarify the contribution of the proposed 4D physical grounding to the final planning performance.

MethodNCDACEPTTCComfortPDMS
Qwen2.5VL-3B SFT97.9392.5277.9394.2099.9584.03
Qwen2.5VL-3B SFT + RL98.3495.9376.8295.7499.9885.97
PhysDrive-3B SFT98.7095.2281.0995.8699.9987.52
PhysDrive-3B SFT + RL w/o Grounding98.8497.3083.0296.9599.9789.96
PhysDrive-3B SFT + RL w/ Grounding98.9697.6583.2796.9099.9890.25

As shown in the table above, the Qwen2.5VL-3B baseline achieves a PDMS of 84.03 after SFT, which further improves to 85.97 with RL, yielding a gain of 1.94 points. In contrast, incorporating our 4D physical grounding during SFT improves PDMS to 87.52, outperforming the baseline by 3.49 points without using RL. Furthermore, RL improves the PDMS of PhysDrive-3B from 87.52 to 90.25 (+2.73), a larger gain than that achieved by the baseline (+1.94). These results demonstrate that 4D physical grounding not only improves planning performance during SFT but also provides a stronger foundation for subsequent RL optimization.

Regarding the use of grounding supervision during RL, this design choice was driven by our experimental results. Specifically, we found that retaining the grounding loss during RL yielded slightly better planning performance. As shown in the table, PhysDrive-3B achieves a PDMS of 89.96 without grounding supervision during RL, compared with 90.25 when grounding supervision is retained. Although the improvement is modest (0.29 points), this observation motivated us to retain the grounding loss during RL in our original method.

Thank you for the response, which has addressed most of my concerns. Would love to see those changes in the camera-ready version. I will raise my score.

We sincerely thank the reviewer for the thoughtful review and constructive suggestions. We will carefully incorporate the corresponding revisions and additional experimental results into the camera-ready version if accepted. We greatly appreciate your time and valuable feedback.

Reviewer_s9fNOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

PhysDrive uses occupancy supervision to incorporate 3D information into the internal representations of an autonomous driving pipeline. Compared with approaches that generate future frames, it avoids shortcuts and background noise associated with RGB generation. Dynamic token constraints are used to constrain geometric information within the vision-language model (VLM), and a two-stage training procedure further improves model performance.

Critical strengths

The use of 3D supervision explicitly requires the model to learn internal 3D representations and predict future occupancy. This has strong practical relevance to collision-related metrics.

Critical weaknesses

Although the paper proposes a two-stage training procedure, the reinforcement learning (RL) used in the second stage appears insufficiently connected to the paper’s central premise. The paper introduces 3D supervision, but the visualizations suggest that its predictions of the road ahead may not be sufficiently accurate. Could these inaccuracies lead to additional collisions?

Questions for authors

Could the authors explain why RL is necessary and provide evidence supporting its role in the proposed approach? This would help assess whether the second training stage is substantively connected to the paper’s central premise. Could the authors provide visual demonstrations showing how future occupancy predictions influence the model’s planning predictions? Such evidence would help assess the relevance of occupancy prediction to planning and clarify the concern raised by the apparent inaccuracies in the visualizations.

2 comments

We thank the reviewer for recognizing the contributions of our work. Below, we clarify the motivation for introducing RL and explain how future occupancy supervision influences trajectory planning.

Q1. Why is RL necessary?

We introduce GRPO-based RL to further improve trajectory generation, as text-based planning still faces limitations despite the enhanced representations learned through our 4D physical grounding. To unify driving understanding and planning, we adopt an autoregressive language decoder for both tasks. However, unlike regression-based decoders that directly minimize numerical trajectory errors, next-token prediction (NTP) provides indirect supervision over tokenized coordinates. Moreover, NTP primarily encourages imitation without explicitly optimizing driving-oriented objectives such as safety, comfort, and progress. Therefore, GRPO further aligns trajectory generation with these objectives through driving-oriented rewards, aiming to better leverage the physically grounded representations for planning.

To further demonstrate the complementary roles of physical grounding and RL, we conduct controlled experiments using the same Qwen2.5-VL-3B backbone, as summarized in the table below. The baseline without physical grounding achieves a NAVSIM PDMS of 84.03 after SFT, which improves to 85.97 (+1.94) with GRPO. Notably, this remains below the 87.52 achieved by PhysDrive with SFT alone, indicating that improving trajectory generation through RL cannot fully compensate for the absence of physical grounding. Building upon the physically grounded representations, GRPO further improves PhysDrive's PDMS from 87.52 to 90.25 (+2.73). Moreover, the RL improvement is larger for PhysDrive than for the baseline (+2.73 vs. +1.94), suggesting that physically grounded representations provide a stronger foundation for subsequent policy optimization. Together, these results highlight that effective planning benefits from both informative physical representations and well-aligned trajectory generation, with the former providing a stronger foundation for the latter.

MethodPDMS ↑
Qwen2.5-VL-3B SFT84.03
Qwen2.5-VL-3B SFT + RL85.97
PhysDrive-3B SFT87.52
PhysDrive-3B SFT + RL90.25

Q2-1. Visual Demonstration of How Future Occupancy Modeling Influences Planning

We add a new figure in Appendix A.4 (Figure 8) to visualize how future occupancy predictions relate to trajectory planning. In fact, 4D occupancy is mainly used to supervise the VLM's internal representations and dynamic latent tokens during training. During inference, the VLM first generates the dynamic latent tokens and then predicts the planning waypoints. Future occupancy can be optionally decoded but is not required for planning. As illustrated in Figure 8, the dynamic latent tokens play a central role in both future scene modeling and trajectory generation, providing shared information that influences both future occupancy predictions and planning waypoints.

Q2-2. Inaccuracies in Occupancy Predictions

We agree with the reviewer that our occupancy predictions contain certain inaccuracies. However, our model is not primarily designed for occupancy prediction, but rather to learn physically grounded representations through 4D occupancy supervision. While the decoded occupancy predictions may not be perfectly accurate, they are neither fed back into the VLM nor used as explicit inputs for trajectory generation. Consequently, occupancy prediction errors do not introduce an additional feedback loop that could lead to error accumulation during planning.

We sincerely appreciate the reviewer's positive assessment and recognition of our work. We look forward to your feedback. Thank you again for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.