acceptodds
Under review as a conference paper at ICLR 2027

RaPhy-AD: Radar-Aware Foundation Model with Explicit Physical Chain-of-Thought for Autonomous Driving

Abstract

Vision-language models (VLMs) have enabled autonomous driving systems to predict trajectories through intermediate reasoning steps. However, VLM-based reasoning is often driven by visual appearance rather than grounded in the physical quantities underlying the scene, such as the distance and relative motion of surrounding agents. Visually driven reasoning can generate linguistically plausible rationales that still remain inconsistent with the actual physical state of the scene, creating a physical-grounding gap between inferred reasoning and directly measured physical quantities. To bridge this gap, we propose RaPhy-AD, a radar-aware foundation model that extends VLM perception beyond visual by incorporating millimeter-wave radar and generates structured physical rationales grounded in radar measurements of object range, azimuth, and Doppler-based radial velocity. To represent these radar-derived physical quantities, RaPhy-AD integrates a Physical-Aware Tokenizer for numerical values and physical units and a Polar-Time Radar Encoder that preserves the temporal, azimuth, and range structure of multi-frame radar observations. We further develop a two-stage training strategy consisting of radar-aware supervised fine-tuning followed by Group Relative Policy Optimization with verifiable physical rewards to first learn radar-grounded physical knowledge and then refine the numerical accuracy of physical reasoning and trajectory prediction. RaPhy-AD shows strong trajectory prediction performance across challenging driving scenarios, particularly under visual degradation, highlighting the importance of radar-grounded physical information.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Autonomous Driving, Vision-Language-Action Models, Automotive Radar, Multimodal Foundation Models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_31UhOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

RaPhy-AD incorporates radar observations into a pretrained vision-language model to generate structured physical rationales before predicting future ego trajectories. It introduces a Physical-Aware Tokenizer that represents numerical values together with their units, and a Polar-Time Radar Encoder that captures spatial and temporal radar structure. Training combines multitask supervised fine-tuning with GRPO using verifiable rewards for trajectories and physical quantities.

Critical strengths

he paper provides quantitative evidence that generated physical descriptions use scene-specific radar information. Beyond qualitative examples, Tables 20–21 evaluate physical quantities and rationale–trajectory consistency. Replacing the original radar with mismatched radar increases closing-velocity MAE from 0.712 to 3.440 m/s and trajectory L2 from 0.144 to 0.450 m (Table 5). These results support the relevance of radar to the model’s predictions and make the physical-grounding contribution more convincing.

Critical weaknesses

Appendix F.1 states that RaPhy-AD receives an oracle LEFT/RIGHT/STRAIGHT command derived from the future ground-truth trajectory, whereas navigation prompting is disabled for Alpamayo 1.5 despite being supported. This difference can affect trajectory prediction, particularly in turning and lane-change scenarios, and makes it difficult to attribute the reported advantage to radar and the proposed reasoning method. Comparisons under equivalent route conditioning are needed to establish the magnitude of the contribution. The necessity and sensitivity of this alignment procedure should be clarified.

The SFT comparison provides initial evidence for rationale supervision, but there is no matched GRPO control that predicts trajectories without generating rationales. Trajectory and endpoint rewards together receive a weight of 0.66 in Eq. 17, so direct trajectory optimization remains a plausible explanation for much of the final improvement. Improvements in both rationale quality and trajectory accuracy do not by themselves distinguish these effects. Furthermore, the tokenizer and encoder ablations use only 100 optimization steps and operate at substantially higher errors than the final model. They establish benefits during early optimization, but leave their contribution after full training uncertain. These limitations weaken the evidence for the paper’s central methodological claims.

Questions for authors

See weaknesses.

3 comments

We thank the reviewer for recognizing the value of grounding driving predictions in scene-specific radar measurements and for highlighting the quantitative evidence supporting our physical-grounding design. We also thank the reviewer for raising important concerns regarding route-conditioning fairness, the contribution of rationale supervision beyond direct trajectory optimization, and whether the gains from our tokenizer and radar encoder persist after full training. We have added the corresponding experiments and analyses, which we discuss below.

W1. Appendix F.1 states that RaPhy-AD receives an oracle LEFT/RIGHT/STRAIGHT command derived from the future ground-truth trajectory, whereas navigation prompting is disabled for Alpamayo 1.5 despite being supported. This difference can affect trajectory prediction, particularly in turning and lane-change scenarios, and makes it difficult to attribute the reported advantage to radar and the proposed reasoning method. Comparisons under equivalent route conditioning are needed to establish the magnitude of the contribution. The necessity and sensitivity of this alignment procedure should be clarified.

W1. Evaluation under Matched Route Conditioning

We would first like to clarify that in Table 2 of our paper, RaPhy-AD, Alpamayo 1, and Alpamayo 1.5 were all provided with the same oracle LEFT/RIGHT/STRAIGHT route command.

  • Appendix F.1 did not clearly distinguish between providing a route command through the text prompt and enabling a dedicated navigation-conditioning mechanism:
    • Alpamayo 1: Since it has no dedicated route-conditioning interface, we provided the oracle route command directly through the text prompt.
    • Alpamayo 1.5: We provided the same oracle route command through the text prompt but disabled its optional special-token-based navigation conditioning, following the official recommendation that this option may degrade trajectory quality.

We apologize for the ambiguity in our description and will revise Appendix F.1 to explicitly clarify how route commands were provided to each model and which navigation-conditioning option was disabled.

Nevertheless, we agree with the reviewer that differences in route conditioning can make it difficult to attribute the reported gains to radar and the proposed reasoning method. To address this concern, we additionally compare RaPhy-AD with Alpamayo 1 and Alpamayo 1.5 under a matched setting in which none of the models receives a route command.

ModelTrajectory L2 (m) ↓Collision Rate (%) ↓
1s2s3sAvg.1s2s3sAvg.
Alpamayo 10.1040.3320.6850.3740.0000.0000.0900.030
Alpamayo 1.50.1080.3940.8730.4580.0000.0190.1680.062
RaPhy-AD (w/o route command)0.0880.1830.3610.2110.0000.0000.0900.030

Interpretation: Even under this stricter setting, where none of the models receives a route command, RaPhy-AD reduces the average trajectory L2 error by 43.6% relative to Alpamayo 1 and 53.9% relative to Alpamayo 1.5. The difference becomes even more pronounced at the 3s horizon, with reductions of 47.3% and 58.6%, respectively. RaPhy-AD also matches Alpamayo 1 in average collision rate (0.030%) and reduces the collision rate by 51.6% relative to Alpamayo 1.5 (0.062% → 0.030%). These results further support the effectiveness of our radar-grounded approach even in the absence of explicit route commands.

W2. The SFT comparison provides initial evidence for rationale supervision, but there is no matched GRPO control that predicts trajectories without generating rationales. Trajectory and endpoint rewards together receive a weight of 0.66 in Eq. 17, so direct trajectory optimization remains a plausible explanation for much of the final improvement. Improvements in both rationale quality and trajectory accuracy do not by themselves distinguish these effects.

W2. Disentangling Trajectory Rewards from Rationale Supervision

We agree that improvements in both trajectory accuracy and rationale quality alone do not isolate whether the final gain comes from direct trajectory optimization or from the explicit physical reasoning process. To disentangle these effects, we conduct two complementary controls: (1) trajectory-only GRPO, which removes all rationale-related rewards while retaining direct trajectory optimization, and (2) rationale-free trajectory prediction, which removes explicit rationale generation while preserving the same downstream trajectory objective.

1. Is direct trajectory reward sufficient?

We first compare our full GRPO objective against trajectory/endpoint-only GRPO under the same training setting.

Trajectory MetricsRationale Metrics
TrainingTraj. L2 Avg. ↓Collision Rate (%) ↓Range MAE ↓Velocity MAE ↓Ego +1s Speed MAE
SFT0.3000.0731.0330.7620.212
+ Trajectory-only GRPO0.2870.0641.1780.8020.285
+ Full GRPO (Ours)0.1440.0140.7620.6540.126

Interpretation. Trajectory-only GRPO provides only a modest improvement over SFT in trajectory metrics, while degrading the numerical accuracy of the physical rationale. In contrast, Full GRPO substantially improves both planning performance and physical-state accuracy. This indicates that the large final gain cannot be explained by direct trajectory rewards alone; the rationale-related rewards provide an additional learning signal that is important for downstream planning.

2. Is explicit rationale generation itself necessary?

We next remove the explicit rationale-generation process and directly predict the trajectory from the model representation. We additionally consider a structured-physical-states-only variant to distinguish explicit reasoning from merely exposing intermediate physical quantities. The rationale-free direct-trajectory control uses the same backbone, radar inputs, and training data as Full Rationale, with a matched SFT/GRPO optimization budget.

TrainingTraj. L2 Avg↓Collision Rate (%) ↓
Direct Trajectory prediction (w/o rationale)0.2780.021
structured physical states alone0.3780.095
Full Rationale (Ours)0.1440.014

Interpretation. Removing explicit rationale generation substantially degrades trajectory accuracy, even when the model is still optimized for the same final planning objective. Moreover, simply exposing structured physical states is insufficient to recover the performance of the full model. Together with the trajectory-only GRPO result above, this suggests that RaPhy-AD benefits not only from direct trajectory optimization, but also from explicitly generating and supervising the intermediate physical reasoning process.

W3. Furthermore, the tokenizer and encoder ablations use only 100 optimization steps and operate at substantially higher errors than the final model. They establish benefits during early optimization, but leave their contribution after full training uncertain. These limitations weaken the evidence for the paper’s central methodological claims.

W3. Full-Training Validation of Tokenizer and Radar Encoder Ablations

We thank the reviewer for emphasizing that the original 100-step ablations mainly demonstrate early-stage optimization behavior and do not, by themselves, establish the contribution of the Physical-Aware Tokenizer and Polar-Time Radar Encoder after full training. We agree that this is an important limitation of the original Table 6, especially given that these components constitute central methodological elements of RaPhy-AD.

  1. Why the original ablations were reported at 100 steps. During training, several alternative configurations exhibited a rapid slowdown in improvement at an early stage, whereas the proposed Physical-Aware Tokenizer and Polar-Time Radar Encoder continued to reduce error as optimization progressed. To compare all variants under the same computational budget, we therefore reported the matched 100-step checkpoint in the original Table 6.
  2. Limitation of the original evidence. The original experiment was intended to compare early learning efficiency and convergence behavior rather than to claim full-training superiority. Because the alternative variants were not trained through the complete schedule at the time of submission, the 100-step results alone could not establish whether the observed gains would persist at the much lower error regime reached by the final model.
  3. Full-training verification under matched budgets. To directly address this concern, we now extend all tokenizer and radar-encoder variants through the matched SFT and GRPO schedules using the same data and compute budgets, and report their performance throughout training below.

Tokenizer

VariantSFT 10050010002000300040004300GRPO 1005001200
Standard1.3821.2100.9940.8510.8070.8640.8510.8960.8570.802
Atomic Numeric1.2151.1040.9650.8010.8370.8430.7540.8020.7390.755
Unit-Integrated1.2140.9420.6010.5240.5470.5200.5120.6020.5870.589
Atomic+Unit1.1350.7870.3510.3260.3010.2760.2690.2540.2030.144

Radar Encoder

SFTGRPO
Variant \ Step100500100020003000400043001005001200
Global1.3281.1350.8940.8820.8700.8580.8540.8420.8290.813
Frame1.1911.0460.8650.8370.8080.7800.7710.7750.7490.751
Polar1.2140.9360.5890.5930.5970.6010.6020.5930.5880.556
Frame + Polar1.1760.8170.3680.3380.3080.2780.2690.2430.2310.144

Interpretation. The full-training results confirm that the 100-step trends are not limited to early optimization. For the tokenizer, Standard and Atomic Numeric begin to plateau after roughly 2,000–3,000 steps, while Unit-Integrated largely saturates after 1,000–2,000 steps; in contrast, Atomic+Unit continues to improve through the full SFT schedule and reaches 0.144 after GRPO. Similarly, for the radar encoder, Global and Polar largely plateau after around 1,000 steps and Frame shows only limited subsequent improvement, whereas Frame+Polar continues to improve throughout training and reaches the same 0.144 final error.

These results therefore strengthen the central methodological claim: the benefits of the proposed tokenizer and radar encoder are not confined to the high-error, early-training regime, but persist—and become more pronounced—under the matched full-training budget.

Reviewer_PxSXOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

RaPhy-AD first predicts structured text and then produces a plan. VLM reasoning can generate language that appears plausible but is inconsistent with the actual physical state. To address this issue, the paper decomposes 3D physical information into structured text and uses text supervision to require the model to output interpretable physical signals that support planning. Radar provides distance and velocity information. The method achieves competitive results on several mainstream benchmarks.

Critical strengths

Beyond incorporating radar, the authors design multitask data, radar spatiotemporal encoding, and an output interface for physical quantities. The radar zeroing and mismatching experiments indicate that the model uses this information. This provides the community with a strong approach to incorporating radar.

Critical weaknesses

Although the authors explain their choices regarding cameras, LiDAR, and radar, they do not appear to evaluate a combined approach. It is unclear what drawbacks would arise from providing all of these inputs to the model simultaneously. During RL, trajectory-related terms appear to receive large weights, while radar-related and structured-text terms receive relatively smaller weights. In particular, the weight for citing physical quantities in the rationale is only 0.02. This raises the question of whether the performance gains primarily come from directly rewarding trajectory accuracy, rather than from improved reasoning ability.

Questions for authors

To more convincingly establish RL or supervised fine-tuning (SFT) for physical chain-of-thought (CoT) as a core contribution, could the authors compare rewarding trajectories alone with rewarding both trajectories and structured rationales? Could they also provide a clearer analysis of the effects of different reward weights? These comparisons would help determine the contribution of physical reasoning beyond direct trajectory rewards. The largest performance gains appear to occur during RL, while the relative gains from adding rationales seem small, and the benefit of structured physical information during RL is not evident. Moreover, the zero-shot results on nuScenes do not appear to show consistent improvements in both L2 and collision rate after RL. Could the authors provide evidence clarifying whether RL primarily fits the original data distribution or enables the model to learn physical information? This distinction would affect the assessment of whether the RL gains reflect improved physical reasoning.

5 comments

We thank the reviewer for recognizing the value of our radar-aware design, including the multitask training data, spatiotemporal radar encoding, and explicit physical-quantity interface, as well as the evidence that RaPhy-AD actively uses radar information. We also thank the reviewer for raising important questions regarding the benefit and cost of combining camera, LiDAR, and radar, and whether the gains from RL primarily arise from direct trajectory optimization or from improved physical reasoning. We address these points with additional sensor-combination and reward-ablation experiments below.

W1. Although the authors explain their choices regarding cameras, LiDAR, and radar, they do not appear to evaluate a combined approach. It is unclear what drawbacks would arise from providing all of these inputs to the model simultaneously.

W1. Sensor Combination Ablation

We thank the reviewer for raising this question. To directly examine whether jointly providing all sensing modalities further improves RaPhy-AD, we additionally introduce a LiDAR encoder based on Point Transformer v1 and evaluate four sensor configurations: Camera, Camera+LiDAR, Camera+Radar, and Camera+LiDAR+Radar under the same downstream framework.

  • Sensor combination ablation
SensorsL2 1s ↓2s ↓3s ↓Avg. ↓Collision (%) ↓Range MAE ↓Velocity MAE ↓Closing-Vel. MAE ↓
Camera0.1510.6691.6570.8260.614.193.523.16
Camera + LiDAR0.1490.6581.6400.8160.513.543.162.83
Camera + Radar0.1410.6301.5630.7780.513.632.151.80
Camera + LiDAR + Radar0.1400.6281.5800.7810.523.352.131.79

The results show that LiDAR and radar provide complementary information. LiDAR primarily improves geometric accuracy, reducing range MAE from 4.19 m to 3.54 m, whereas radar yields substantially larger improvements in motion estimation, reducing velocity MAE from 3.52 to 2.15 m/s and closing-velocity MAE from 3.16 to 1.80 m/s. Combining all three modalities further improves physical-state estimation, particularly range, but provides almost no additional trajectory improvement over Camera+Radar (0.781 vs. 0.778 m average L2).

This marginal trajectory gain comes with a substantially larger token budget. With 10 sensor frames, Camera+Radar produces 960 input tokens, whereas Camera+LiDAR+Radar produces 2,240 tokens, corresponding to approximately 2.3× more tokens. Thus, while LiDAR provides complementary geometric information, jointly using all modalities substantially increases multimodal input complexity for only marginal trajectory improvement.

We further investigate the effect of temporal context by progressively increasing the number of LiDAR and radar frames from 1 to 10.

  • Motion estimation under different temporal contexts
SensorFramesTemporal ContextVelocity MAE ↓Closing-Vel. MAE ↓
LiDAR10 s3.232.87
Radar10 s2.141.79
LiDAR20.1 s3.282.91
Radar20.1 s2.121.74
LiDAR50.4 s3.202.84
Radar50.4 s2.081.74
LiDAR100.9 s3.102.74
Radar100.9 s2.111.78

Notably, radar already achieves a velocity MAE of 2.14 m/s and a closing-velocity MAE of 1.79 m/s using only a single frame, while LiDAR remains less accurate even with 10 frames (3.10 and 2.74 m/s, respectively). Radar performance also remains stable as the temporal context increases. This supports our motivation for using radar: unlike LiDAR, which primarily provides geometry and requires temporal observations to infer motion, radar directly provides Doppler-based motion information from a single observation.

Overall, these results show that LiDAR and radar are complementary, but also clarify the design choice of RaPhy-AD. LiDAR provides stronger geometric measurements, whereas radar provides substantially more efficient motion information with short temporal context. RaPhy-AD therefore focuses on radar as an efficient source of both spatial and explicit motion cues for physically grounded reasoning.

W2. During RL, trajectory-related terms appear to receive large weights, while radar-related and structured-text terms receive relatively smaller weights. In particular, the weight for citing physical quantities in the rationale is only 0.02. This raises the question of whether the performance gains primarily come from directly rewarding trajectory accuracy, rather than from improved reasoning ability.

Q1. To more convincingly establish RL or supervised fine-tuning (SFT) for physical chain-of-thought (CoT) as a core contribution, could the authors compare rewarding trajectories alone with rewarding both trajectories and structured rationales? Could they also provide a clearer analysis of the effects of different reward weights? These comparisons would help determine the contribution of physical reasoning beyond direct trajectory rewards.

W2 & Q1. Isolating the Contribution of Physical-Reasoning Rewards

We thank the reviewer for the careful observation. We agree that the role of our physical-reasoning rewards, relative to the more heavily weighted trajectory objectives, deserves further clarification. We first clarify two aspects of our reward design.

  1. All non-trajectory/endpoint rewards directly supervise the structured physical rationale. Rewards for risk, object states, ego states, visual cues, numerical consistency, and other physical reasoning components collectively account for 34% of the total reward. Thus, although trajectory and endpoint terms receive the largest individual weights, a substantial portion of the GRPO objective explicitly supervises the physical and numerical correctness of the intermediate rationale.
  2. The reward weights are designed for the SFT-to-GRPO training stage. SFT first establishes the overall structure of the physical rationale, while its numerical and physical accuracy remains imperfect. GRPO then ranks multiple sampled responses according to relative rewards. We therefore assign larger weights to trajectory and endpoint accuracy to keep optimization aligned with the final driving objective, while the remaining rewards refine the physical and numerical correctness of the structured rationale.

To directly address the reviewer’s central question—whether the gains primarily arise from directly rewarding trajectory accuracy—we compare trajectory/endpoint-only GRPO with our full reward design under the same training setting.

Comparison of trajectory-only GRPO

TrainingTraj. L2 Avg ↓Collision Rate (%) ↓Range MAE ↓Velocity MAE ↓Ego +1s Speed MAE ↓
SFT0.3000.0731.0330.7620.212
+ Trajectory-only GRPO0.2870.0641.1780.8020.285
+ Full GRPO (Ours)0.1440.0140.7620.6540.126

Effect of reward weighting. The trajectory and endpoint terms account for 66% of the total reward, while the remaining 34% is distributed across physical-reasoning components. Our trajectory-only ablation can be viewed as the extreme case in which this balance is shifted entirely toward the final driving objective. Despite receiving direct trajectory supervision, this setting provides only marginal trajectory gains and degrades physical-rationale accuracy. In contrast, retaining the reasoning-related rewards substantially improves both planning and rationale metrics, indicating that the performance gain depends on balancing trajectory optimization with physical-reasoning supervision rather than simply increasing the weight on trajectory accuracy.

Q2. Moreover, the zero-shot results on nuScenes do not appear to show consistent improvements in both L2 and collision rate after RL. Could the authors provide evidence clarifying whether RL primarily fits the original data distribution or enables the model to learn physical information? This distinction would affect the assessment of whether the RL gains reflect improved physical reasoning.

Q2. Zero-Shot Generalization of Physical Reasoning

We thank the reviewer for raising this important question. We agree that the inconsistent zero-shot gains on nuScenes require distinguishing between overfitting to the source-domain distribution and learning transferable physical information. Our additional analysis suggests that the limited gain on nuScenes is strongly affected by the substantial radar-sensor domain shift between the two datasets. Radar quality differs substantially between the source and target domains.

Radar StatisticsnuScenesNVIDIA
Radar points per frame (median)61≥656
Front objects with no radar return48.0%28.9%
Radar points per object0.944.20

Compared with the high-resolution radar used in our training data, nuScenes radar is substantially sparser. Nearly half of the front objects in nuScenes receive no radar return, and even detected objects contain fewer than one radar point on average. This significantly limits the physical evidence available to a radar-grounded reasoning model at test time.

To further separate this sensor-domain effect from model fitting, we analyze zero-shot nuScenes performance according to the radar quality of each scene.

Radar Quality (points/object)AnchorsObjects Hit by RadarL2 w/ Radar ↓L2 w/o Radar ↓Radar Contribution ΔL2 [95% CI]Relative GainCollision (%) w/ → w/o Radar
Very low (<0.5)97525%0.4860.521+0.036 [+0.014, +0.060]+7.4%0.07 → 0.16
Low (0.5–1.0)1,49548%0.5090.553+0.044 [+0.020, +0.069]+8.7%0.24 → 0.33
Medium (1.0–1.5)1,39665%0.4760.549+0.072 [+0.047, +0.097]+15.1%0.12 → 0.23
High (≥1.5)1,04576%0.4730.563+0.090 [+0.055, +0.124]+18.9%0.18 → 0.36

The radar contribution increases monotonically with radar quality, from +7.4% in the lowest-quality group to +18.9% in the highest-quality group. The difference in radar contribution between high-quality scenes (≥1.0 point/object) and very low-quality scenes (<0.5 point/object) is +0.044 m [95% CI: +0.012, +0.076], indicating a statistically significant dependence on the availability of usable radar measurements.

We observe the same trend when radar quality is defined by the fraction of objects successfully receiving radar returns: the radar contribution increases from +0.030 m to +0.071 m as this fraction increases, with a significant high-vs.-low difference of +0.040 m [95% CI: +0.002, +0.074].

Interpretation. Within the same nuScenes distribution, the benefit of radar-grounded reasoning becomes substantially larger when reliable radar measurements are available. This radar-quality trend provides evidence of transferable use of physical radar information and suggests that the transfer gain is constrained by the much sparser and lower-quality radar observations in nuScenes. However, this analysis does not fully isolate how much of this behavior is introduced specifically by GRPO rather than already acquired during SFT; a matched SFT-versus-GRPO comparison within the same radar-quality groups would be needed to distinguish these contributions.

Reviewer_R45GOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The motivation and overall architecture are clearly presented. The appendices provide useful implementation and evaluation details. However, the route-conditioning differences between methods should be disclosed more prominently alongside the main results.

Critical strengths

The method addresses a meaningful limitation of driving VLMs: explanations can be linguistically plausible without accurately representing distances and relative motion. Integrating radar measurements, explicit numerical representations, and verifiable supervision provides a coherent approach to this problem.

The radar perturbation experiments provide useful evidence that the model uses radar information. Average trajectory error increases from 0.144 m with correct radar to 0.323 m with zero radar and 0.450 m with mismatched radar (Table 5). Appendix F additionally evaluates physical accuracy: GRPO reduces object-range MAE from 1.033 to 0.762 m. These diagnostics make the empirical analysis more informative than trajectory metrics alone.

Critical weaknesses

The headline comparison does not isolate the proposed method’s benefits. Appendix F.1 states that RaPhy-AD receives a route command derived from the ground-truth future trajectory, whereas Alpamayo 1 lacks this interface and Alpamayo 1.5 is evaluated with navigation conditioning disabled. Oracle route conditioning is a legitimate task setting, but this asymmetry confounds attribution of the reported improvement to radar or physical reasoning. Matched route-conditioned comparisons and a no-command RaPhy-AD evaluation are needed.

The incremental methodological contribution is insufficiently isolated. AutoDrive-R² already combines CoT supervision, GRPO, and physics-grounded rewards; novelty therefore depends on the specific radar representation and numerical-rationale supervision. However, component ablations use only 100 optimization steps (Table 6), leaving their benefits after full training unclear. Moreover, trajectory and endpoint terms account for 66% of the reward weight (Eq. 17), while a matched trajectory-only GRPO control is not reported. Radar perturbations establish reliance on radar but do not isolate the causal contribution of explicit rationales. The unchanged object F1 and hallucination rate after GRPO (Table 20) further limit claims about improved reasoning reliability.

Questions for authors

Can you report RaPhy-AD without oracle route commands and compare against Alpamayo 1.5 with equivalent navigation information? How much of the reported improvement remains under matched conditioning?

Do the tokenizer and encoder improvements persist after full training under matched data and compute budgets?

Under the same backbone, radar inputs, and training budget, how does the full reward compare with trajectory/endpoint-only GRPO? Would generating structured object states without a free-form driving rationale retain the improvement?

Please clarify the specific methodological advances over AutoDrive-R², and discuss the relationship to MoRAL and RCVLA, taking their public-release dates and possible concurrent development into account.

5 comments

We thank the reviewer for recognizing the importance of accurately representing distances and relative motion in driving VLMs, and for appreciating our integration of radar measurements and explicit numerical representations. We also thank the reviewer for raising important questions regarding route-conditioning fairness, full-training component ablations, the contributions of our reward design and explicit rationales, and our methodological distinctions from related work. We have added the corresponding experiments and analyses, which we discuss below.

W1. The headline comparison does not isolate the proposed method’s benefits. Appendix F.1 states that RaPhy-AD receives a route command derived from the ground-truth future trajectory, whereas Alpamayo 1 lacks this interface and Alpamayo 1.5 is evaluated with navigation conditioning disabled. Oracle route conditioning is a legitimate task setting, but this asymmetry confounds attribution of the reported improvement to radar or physical reasoning. Matched route-conditioned comparisons and a no-command RaPhy-AD evaluation are needed.

Q1. Can you report RaPhy-AD without oracle route commands and compare against Alpamayo 1.5 with equivalent navigation information? How much of the reported improvement remains under matched conditioning?

W1 & Q1. Evaluation under Matched Route Conditioning

We first clarify that oracle route commands were also provided to Alpamayo 1 and 1.5 through their text prompts in our original comparison. We acknowledge that our description in Appendix F.1, “Alpamayo 1 lacks this interface and Alpamayo 1.5 is evaluated with navigation conditioning disabled,” was insufficiently clear.

Specifically,

  • Alpamayo 1 does not provide a dedicated encoder or interface for route commands; its official evaluation instead considers settings with and without route information. We therefore provide the route command directly through the text prompt.
  • Alpamayo 1.5 provides an optional navigation-conditioning mechanism that encloses the command with dedicated tokens, allowing the model to attend more strongly to the navigation instruction. However, its official documentation notes that strongly enforcing this option may degrade trajectory quality. We therefore disabled this dedicated navigation-conditioning option while still providing the same route command through the text prompt.

Nevertheless, to explicitly examine the effect of route commands and address the reviewer’s concern regarding matched conditioning, we additionally evaluate RaPhy-AD without the route command:

ModelTrajectory L2 (m) ↓Collision Rate (%) ↓
1s2s3sAvg.1s2s3sAvg.
Alpamayo 10.1040.3320.6850.3740.0000.0000.0900.030
Alpamayo 1.50.1080.3940.8730.4580.0000.0190.1680.062
RaPhy-AD0.0880.1830.3610.2110.0000.0000.0900.030

Even without the route command, RaPhy-AD reduces the average trajectory error by 43.6% and 53.9% compared with Alpamayo 1 and 1.5, respectively. The improvement becomes more pronounced at longer horizons: at 3s, RaPhy-AD reduces trajectory error by 47.3% and 58.6%, respectively. RaPhy-AD also achieves the same average collision rate as Alpamayo 1 (0.030) and a 51.6% lower rate than Alpamayo 1.5 (0.062 → 0.030). These results show that RaPhy-AD’s improvement persists even without oracle route conditioning.

W2. The incremental methodological contribution is insufficiently isolated. AutoDrive-R² already combines CoT supervision, GRPO, and physics-grounded rewards; novelty therefore depends on the specific radar representation and numerical-rationale supervision. However, component ablations use only 100 optimization steps (Table 6), leaving their benefits after full training unclear.

Q2. Do the tokenizer and encoder improvements persist after full training under matched data and compute budgets?

W2 & Q2. Full-Training Component Ablations

We thank the reviewer for highlighting the importance of validating whether the gains from our Physical-Aware Tokenizer and Polar-Time Radar Encoder persist beyond the limited 100-step ablation setting. To clarify this point, we first explain why all configurations in the original Table 6 were reported at 100 optimization steps and the intended purpose of this experiment.

  1. Why we reported 100-step results. During training, the alternative component configurations showed a rapid slowdown in loss reduction at an early stage, whereas the proposed Physical-Aware Tokenizer and Polar-Time Radar Encoder continued to improve with further optimization. For a fair comparison under the same computational budget, we therefore reported all variants at the matched 100-step checkpoint in the original Table 6.
  2. Limitation of the original setting. The purpose of Table 6 was to compare the early-stage learning efficiency and convergence behavior of the different component configurations. However, because the alternative configurations were not trained through the full schedule at the time of submission, the original results alone cannot fully demonstrate whether the observed advantages persist after full training.
  3. Additional full-training evaluation. We fully agree with the reviewer that this should be verified under a matched full-training setting. We therefore extend the training of the component variants under matched data and compute budgets and report the resulting performance trends below.
  • Tokenizer
SFT stepGRPO step
Variant100500100020003000400043001005001200
Standard1.3821.2100.9940.8510.8070.8640.8510.8960.8570.802
Atomic Numeric1.2151.1040.9650.8010.8370.8430.7540.8020.7390.755
Unit-Integrated1.2140.9420.6010.5240.5470.5200.5120.6020.5870.589
Atomic+Unit1.1350.7870.3510.3260.3010.2760.2690.2540.2030.144
  • Radar Encoder
SFT stepGRPO step
Variant100500100020003000400043001005001200
Global1.3281.1350.8940.8820.8700.8580.8540.8420.8290.813
Frame1.1911.0460.8650.8370.8080.7800.7710.7750.7490.751
Polar1.2140.9360.5890.5930.5970.6010.6020.5930.5880.556
Frame + Polar1.1760.8170.3680.3380.3080.2780.2690.2430.2310.144

For the tokenizer ablation, Standard and Atomic Numeric begin to plateau after roughly 2,000–3,000 steps, while Unit-Integrated largely saturates after 1,000–2,000 steps; in contrast, Atomic+Unit continues to improve through 4,300 steps. Similarly, for the radar encoder, Global and Polar largely plateau after around 1,000 steps and Frame shows only limited subsequent improvement, whereas Frame+Polar continues to improve throughout training.

W3. Moreover, trajectory and endpoint terms account for 66% of the reward weight (Eq. 17), while a matched trajectory-only GRPO control is not reported. Radar perturbations establish reliance on radar but do not isolate the causal contribution of explicit rationales. The unchanged object F1 and hallucination rate after GRPO (Table 20) further limit claims about improved reasoning reliability.

Q3. Under the same backbone, radar inputs, and training budget, how does the full reward compare with trajectory/endpoint-only GRPO? Would generating structured object states without a free-form driving rationale retain the improvement?

W3 & Q3. Isolating the Contributions of Reward Design and Explicit Rationales

We thank the reviewer for the careful observation. We agree that the motivation behind our GRPO training and reward weighting deserves further clarification. We first highlight two key points:

  1. All non-trajectory rewards directly supervise our structured rationale. Specifically, the rewards for risk, object states, ego states, visual cues, and other physical reasoning components collectively account for 34% of the total reward and jointly define the physical and numerical structure of the rationale.
  2. The reward weights are empirically designed for the SFT-to-GRPO training stage. After SFT, the overall rationale structure is already established, but its numerical and physical accuracy remains limited. GRPO then ranks multiple sampled responses according to their relative rewards. We therefore empirically assign larger weights to trajectory and endpoint rewards to keep optimization aligned with the final driving objective, while the remaining rewards refine the physical and numerical accuracy of the structured rationale.

While our reward design intentionally places greater emphasis on trajectory and endpoint accuracy, we appreciate the reviewer for raising an important question: are the physical reasoning rewards still necessary? To directly address this, we additionally compare our full reward design with trajectory/endpoint-only GRPO under the same training setting.

  • Comparison of trajectory-only GRPO
Rationale
TrainingTraj. L2 Avg↓Collision Rate (%) ↓Range MAE ↓Velocity MAE ↓Ego +1s Speed MAE
SFT0.3000.0731.0330.7620.212
+ Trajectory-only GRPO0.2870.0641.1780.8020.285
+ Full GRPO (Ours)0.1440.0140.7620.6540.126

Interpretation: Trajectory-only GRPO provides only marginal trajectory improvement while degrading rationale accuracy, whereas Full GRPO substantially improves both trajectory performance and rationale accuracy.

Beyond the reward design, the reviewer also raises a complementary question: is it necessary to explicitly generate a natural-language rationale, or are structured physical states alone sufficient? Our motivation for explicit rationales is twofold: (1) interpretability, by exposing the physical evidence underlying the driving decision, and (2) reasoning consistency, by explicitly connecting scene-level physical states to trajectory prediction.

Nevertheless, to isolate the contribution of this explicit reasoning process, we additionally compare our full model against a structured-states-only variant that predicts the trajectory directly from the structured physical states without generating the explicit structured rationale.

Comparison of structured physical states alone

TrainingTraj. L2 Avg↓Collision Rate (%) ↓
Structured physical states alone0.3780.095
Full Rationale (Ours)0.1440.014

Interpretation: Using structured physical states alone provides insufficient reasoning support for connecting physical scene information to trajectory prediction, resulting in substantially worse trajectory performance. Explicitly generating the full rationale improves this connection and leads to markedly lower trajectory error and collision rate.

Finally, regarding the unchanged object F1 and hallucination rate after GRPO (Table 20), we note that both metrics had already largely converged during SFT with large-scale automatically generated object-detection supervision. Moreover, since not every frame contains a collision-relevant object, the learning signal for identifying such objects is relatively sparse; GRPO therefore mainly improves the numerical and physical accuracy of the rationale rather than object recognition itself.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.