acceptodds
Under review as a conference paper at ICLR 2027

RCVLA: 4D Radar-Grounded Semantic Reasoning and Trajectory Arbitration for Autonomous Driving

Abstract

4D radar provides geometric and motion cues that complement visual semantics, but integrating it into vision-language-action (VLA) models requires both radar-language alignment for semantic reasoning and explicit use of radar measurements for trajectory refinement and selection. To support these capabilities, we construct Cap4DR with 86,016 radar-image-text samples for alignment pretraining and OmniHD-QA with 520,161 question-answer pairs for instruction tuning across scene description, key-object reasoning, occupancy understanding, and trajectory planning. Building on these datasets, we propose RCVLA, a radar-camera VLA framework consisting of a radar-grounded semantic reasoning stage (RCVLA-Sem) and a trajectory arbitration stage (RCVLA-Phys). RCVLA-Sem performs gated bidirectional interaction between camera and radar tokens for driving question answering and reference trajectory generation, while auxiliary heads provide object and occupancy queries. RCVLA-Phys refines reference-guided trajectory candidates through truncated diffusion conditioned on these queries and cluster-level radar measurements, then calibrates candidate scores using radar-derived time-to-collision risk. On OmniHD-QA, RCVLA-Sem improves CIDEr by 9.92 points and reduces key-object velocity error by relative to OmniDrive. RCVLA-Phys further reduces average L2 error from to and average open-loop collision rate from to relative to RCVLA-Sem. Ablation studies further show that language-aligned radar tokens improve semantic reasoning, while cluster-level radar measurements and risk calibration improve trajectory arbitration. Code will be released.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
4D Radar

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_gAMkOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

1.Dataset: They construct Cap4DR with 86,016 radar-image-text samples for alignment pretraining and OmniHD-QA with 520,161 question-answer pairs for instruction tuning across scene description, key-object reasoning, occupancy understanding, and trajectory planning; 2. Method: They propose RCVLA, a radarcamera VLA framework consisting of a radar-grounded semantic reasoning stage (RCVLA-Sem) and a trajectory arbitration stage (RCVLA-Phys). 3. Result: On OmniHD-QA, RCVLA-Sem improves CIDEr by 9.92 points and reduces key-object velocity error by 21.9% relative to OmniDrive. RCVLA-Phys further reduces average L2 error from 0.348 to 0.259m and average open-loop collision rate from 0.576% to 0.175% relative to RCVLASem.

Critical strengths
  1. Motivation&dataset: They construct two datasets and develop RCVLA to adapt 4D radar into VLA model. Cap4DR contains 86,016 4D radar-image-text samples for radar–language alignment pretraining, with 4D radar relevant entities verified through multi-model cross checking and selective manual review.
  2. Reasonable Architecture&System: RCVLA combines a reference trajectory from semantic reasoning, structured scene queries, and direct 4D radar measurements for trajectory refinement and risk-aware selection.
  3. Improvement on result: Experiments on OmniHD-QA demonstrate improvements in semantic reasoning and open-loop planning over the evaluated baselines.
Critical weaknesses
  1. Limited novelty: This paper is more like the system framework instead of introducing a methodological contribution. Specifically, this system is similar to DiffusionDrive[1] with the radar version.
  2. Like in the First problem, the advantage of using radar is not well illustrated. Without radar,the model obtain the better trajectory result in Table 4, in L2 and Coll result;
  3. All reported results are based on open-loop evaluation, which is insufficient since it can not evaluate something like recovery from planning errors.
  4. The alignment with radar is not well verified since the alignment dataset is build via camera images without radar, and the training procedure is also with image input;

[1] DiffusionDrive: Truncated Diffusion Model for End-to-End Autonomous Driving

Questions for authors
  1. As discussed in the Weaknesses section.
  2. Could the authors provide more results to better demonstrate the contribution of radar.
  3. Could the authors clarify how radar-language alignment is verified, given that the descriptions and entity validation in Cap4DR are mainly based on camera images?
5 comments
AuthorsOpenReview ↗

Additional Radar Ablations, Alignment Verification, and Novelty Clarification

W1: Novelty

We acknowledge that truncated diffusion and trajectory anchors build upon DiffusionDrive. However, RCVLA goes beyond simply incorporating radar into an existing planner. Our main contributions include: (1) radar–language alignment through Cap4DR and RQ-Former; (2) gated camera–radar interaction for semantic reasoning and structured scene queries; and (3) radar-cluster conditioning for trajectory generation and TTC-based risk calibration for candidate selection. These designs enable the complementary use of 4D radar in both semantic understanding and physical planning. We will further clarify these distinctions in the revision.

W2/Q2: Contribution of 4D Radar

We acknowledge that Table 4 evaluates RCVLA-Sem alone: removing radar yields a lower Sem-stage collision rate of 0.391%, compared with 0.576% for the full Sem model. Improved semantic metrics alone do not ensure safer reference trajectories. We therefore report final RCVLA-Phys results from joint radar-component ablation and alternative semantic-stage variants.

Final Planning SettingL2 (m) ↓Collision (%) ↓
w/o radar clusters0.2970.442
w/o risk calibration0.2960.340
w/o radar clusters & risk calibration0.3390.391
w/o RLP0.2870.422
w/o 4D radar in RCVLA-Sem0.2620.329
BEV-Planner0.4290.298
Full RCVLA0.2590.175

Jointly removing radar-cluster conditioning and TTC calibration from RCVLA-Phys increases the average L2 error from 0.259 to 0.339 m and collision rate from 0.175% to 0.391%, while retaining radar-enabled RCVLA-Sem. Radar-cluster conditioning guides candidate generation, whereas TTC calibration adjusts candidate ranking, which may explain the non-monotonic collision rates across ablations. We further evaluate the contribution of radar-informed semantic outputs while keeping radar available in RCVLA-Phys. Removing radar from RCVLA-Sem increases the final collision rate to 0.329%, while removing radar–language pretraining raises it to 0.422%. These comparisons support the contribution of radar to both stages, although the effects of reference trajectories and structured scene features are not separately isolated. Finally, replacing matched radar velocities with GT planar velocities yields a finite-TTC MAE of 0.272 s and changes candidate selection in only 1.327% of frames, with the average collision rate remaining 0.175% under both velocity settings.

W3: Open-Loop Evaluation Limitations

We agree that open-loop evaluation cannot assess closed-loop recovery or guarantee driving safety. Table 6 tests delayed semantic references and thus probes robustness to stale inputs, but is not a closed-loop perturbation or recovery test. We clarify this limitation in the revision.

W4/Q3: Radar–Language Alignment Verification

Camera images are used to construct radar-oriented language supervision, whereas radar features are the input during radar–language pretraining. Cap4DR contains 86,016 synchronized radar–image–text samples. Camera-generated descriptions focus on traffic participants and spatial relations, excluding visual appearance attributes. Three VLMs cross-check extracted entities against the images, with ambiguous cases manually reviewed. During pretraining, the radar encoder and LLM are frozen; RQ-Former learns from radar features and paired descriptions without image inputs.

RCVLA-Sem SettingCIDEr ↑Velocity Error (m/s) ↓
w/o 4D radar98.703.50
w/o RLP101.603.45
Full RCVLA-Sem102.412.82

The gains over removing radar–language pretraining support its downstream semantic and motion benefits, but do not verify explicit radar-point-to-text correspondence. Our approach uses camera-assisted, paired radar–language supervision rather than instance-level radar–text annotations.

Dear Reviewer gAMk,

We sincerely appreciate your constructive feedback. We have addressed your concerns through additional experiments and manuscript revisions, with all changes highlighted in blue.

We hope these efforts have satisfactorily addressed your concerns and welcome any further suggestions.

Best regards, The Authors

Reviewer_gAMkOpenReview ↗

Reply for Authors

Thanks for the additional experiments and explanations. They provide some evidence for contribution of radar, but the radar-language alignment is still not well verified for camera-derived descriptions. Also, the novelty and lack of closed-loop evaluation remain concerns. I would like to maintain my original rating.

Reviewer_rwxHOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents RCVLA, a camera-4D radar VLA framework that combines RCVLA-Sem (Sem) and RCVLA-Phys (Phys). Sem is an OmniDrive-style Q-Former VLM extended with a radar Q-Former and camera–radar token interaction to answer driving questions, and produces object and occupancy queries. Phys is further trained based on the Sem. It is a DiffusionDrive-style truncated-diffusion planner that refines the candidates. The authors also build two datasets. Cap4DR: 86k radar–image–text samples from six radar datasets, used for radar–language pretraining. OmniHD-QA: 520k QA pairs derived from OmniHD-Scenes, used for instruction tuning and evaluation. On OmniHD-QA, RCVLA-Sem improves CIDEr from 92.49 to 102.41 over OmniDrive, and RCVLA-Phys achieves 0.259 m average planning L2 error and 0.175% average open-loop collision rate.

Critical strengths

S1. The use of 4D radar in driving. The main contribution is the use of radar at two levels: language-aligned tokens for semantic reasoning, and cluster-level motion cues for trajectory refinement. It goes beyond simply adding radar tokens to a VLM. They provide useful evidence (Table 3): removing radar-cluster conditioning increases the 3 s collision rate from 0.401% to 0.988%, while removing risk calibration increases it to 0.741%.

S2. Useful datasets. Cap4DR and OmniHD-QA provide samples from six real-world radar datasets and QA pairs with 4D radar observations. These resources could help further research in radar-grounded language understanding and driving.

Critical weaknesses

Paper position and related ablation. Several highly relevant related works without radar design are not discussed in the paper. DriveVLM-Dual (CoRL 2024) combines low-frequency VLM planning with a fast planner; ORION (ICCV 2025) conditions a generative planner on an LLM planning token, and DrivePI (CVPR 2026) integrates an MLLM with occupancy prediction and a diffusion action head. Discussing them would help clarify where RCVLA's novelty lies.

Based on the main claim of complementary 4D radar, the attribution is still unclear to me. Table A3 provides encouraging evidence for the radar-specific components, but radar also enters through the RQ-Former tokens and the object/occupancy decoders. The "w/o 4D radar" setting in Table 4 evaluates RCVLA-Sem only, without training and evaluating Phys based on the radar-free Sem. The paper does not directly quantify radar's total contribution to the final planning performance. A full camera-only comparison, or a joint ablation of radar-cluster conditioning and risk calibration, would help clarify. It is also unclear whether "w/o radar clusters" retains radar-based TTC computation.

Questions for authors

Q1. Radar contribution. Could the authors clarify details of the existing radar ablations? In particular, does “w/o radar clusters” in Table 5 remove only cluster conditioning from the diffusion decoder, or does it also disable radar-based TTC calibration? A matched camera-only evaluation of the complete RCVLA pipeline, or a Phys ablation jointly removing radar-cluster conditioning and TTC calibration would help.

Q2. TTC modeling. How is the planar velocity vector of each radar cluster estimated from Doppler radial-velocity measurements? Is tangential motion estimated or implicitly assumed to be zero? How are cluster footprints determined for the TTC calculation? Clarifying these details is important for the radar-based risk calibration.

If the authors can establish the radar-specific contribution in final planning performance, I would like to raise my recommendation.

4 comments
AuthorsOpenReview ↗

Additional Radar Ablations, TTC Validation, and Novelty Clarification

We thank Reviewer rwxH for the helpful comments. We clarify the distinctions from related work, report a joint radar ablation, and assess TTC sensitivity to radial-velocity approximation

W1: Novelty and Related Work

We acknowledge DriveVLM-Dual, ORION, DrivePI, and DiffusionDrive, and have expanded the related-work discussion. Our contribution is not the slow–fast structure or truncated diffusion itself, but the use of 4D radar across both stages: radar–language pretraining and gated camera–radar interaction for semantic reasoning, followed by radar-cluster-conditioned trajectory generation and TTC-based candidate selection.

W2/Q1: Radar Contribution and Ablations

In Table 5, "w/o radar clusters" disables radar conditioning in the diffusion decoder but retains radar-based TTC calibration. Following the reviewer's suggestion, we additionally remove both components from RCVLA-Phys while keeping radar-enabled RCVLA-Sem unchanged.

SettingL2 (m) ↓Collision (%) ↓
w/o radar clusters0.2970.442
w/o risk calibration0.2960.340
w/o radar clusters & risk calibration0.3390.391
Full RCVLA0.2590.175

Joint removal increases average L2 from 0.259 to 0.339 m and collision from 0.175% to 0.391%. The non-monotonic collision rates may reflect the distinct roles of cluster conditioning, which guides candidate generation, and TTC calibration, which re-ranks existing candidates.

Q2: Radar Velocity, Footprint, and TTC Sensitivity

We cluster radar points using DBSCAN. Cluster centers and compensated Doppler-derived velocity components are aggregated by coordinate-wise medians; axis-aligned footprints use the 5th–95th percentile ranges of point coordinates. Under constant velocity, clusters are extrapolated at 0.5-s intervals over 3 s. Footprint separation and relative closing speed determine TTC, whose minimum calibrates candidate scores. Tangential velocity is not explicitly recovered.

We replace matched cluster velocities with annotated GT planar velocities, the five candidate trajectories, and raw scores unchanged. TTC MAE uses only estimates finite under both conditions within 3 s. Selection Change is the percentage of frames choosing a different trajectory.

Motion TypeTTC MAE (s) ↓Selection Change (%)Radar L2 (m) ↓GT-Vel L2 (m) ↓Radar Coll. (%) ↓GT-Vel Coll. (%) ↓
Longitudinal0.2661.4220.2620.2630.2370.237
Crossing0.3132.1740.3260.3241.4491.449
Turning0.2043.1250.2080.2080.0000.000
Overall0.2721.3270.2590.2590.1750.175

Crossing scene has the highest finite-TTC MAE of 0.313 s, but only 1.327% of frames select a different candidate. Average L2 and collision remain 0.259 m and 0.175% under both velocity conditions. This supports limited selection sensitivity in the evaluated open-loop setting.

Dear Reviewer rwxH,

We sincerely appreciate your constructive feedback. We have addressed your concerns through additional experiments and manuscript revisions, with all changes highlighted in blue.

We hope these efforts have satisfactorily addressed your concerns and welcome any further suggestions.

Best regards, The Authors

Thank you for the detailed response and the additional experiments. My concerns have been addressed. I have raised my score.

AuthorsOpenReview ↗

Thank You for Your Feedback and Reconsideration

Thank you for taking the time to review our response and reconsider your rating. We really appreciate your feedback and support.

Reviewer_zpAdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces RCVLA, a camera–radar driving framework that separates semantic reasoning from trajectory refinement. RCVLA-Sem combines language-aligned image and radar tokens to answer driving questions and predict a reference trajectory, while auxiliary detection and occupancy heads provide structured scene features. RCVLA-Phys refines trajectory candidates through truncated diffusion and uses radar-derived time-to-collision estimates to calibrate their scores. The authors also construct Cap4DR for radar–language alignment and OmniHD-QA for downstream training and evaluation. On OmniHD-QA, the full system improves scene description and velocity estimation, and reduces average open-loop trajectory error and collision rate relative to the semantic planner. Ablations examine the fusion mechanism, refinement conditions, candidate construction, and delayed references.

Critical strengths
  1. The refinement stage has convincing component-level support.** I like that radar contributes both learned features and an explicit motion-based ranking signal. Table 5 shows that removing radar clusters or risk calibration raises average collision rate from 0.175% to 0.442% or 0.340%, respectively. The occupancy and object ablations also help explain where the improvement comes from. The delayed-reference experiments add a useful practical test beyond evaluating only perfectly synchronized semantic predictions.
Critical weaknesses
  1. The physical risk estimate needs more targeted validation. Section 4.3 extrapolates cluster positions using compensated radial-velocity vectors. Radial measurements alone do not identify tangential motion, which matters for crossing vehicles and pedestrians. The paper does not explain whether additional information recovers that component or quantify how this approximation affects TTC estimates. Aggregate gains from risk calibration are encouraging, but they do not establish when the supposedly physical safety signal is reliable or when it could mis-rank candidates.

  2. The planning value of language-aligned radar remains unclear. In Table 4, removing radar or radar–language pretraining actually lowers the semantic planner's collision rate from 0.576% to 0.391%, with similar L2 error. The final planner improves substantially, but Table 5 only removes individual refinement inputs. There is no matched comparison that keeps the physical refinement machinery while replacing the language-based reference with a conventional planner. This leaves open how much of the final planning benefit requires the proposed semantic stage.

Questions for authors
  1. How are cluster velocities reconstructed beyond their radial components? Could you report TTC accuracy and candidate-selection failures separately for crossing, turning, and longitudinal traffic, ideally comparing the current approximation with annotated full planar velocities?

  2. Could you keep the refinement module, supervision, and candidate budget fixed while comparing references from RCVLA-Sem, its version without radar–language pretraining, and a conventional planner? This would clarify whether semantic alignment improves the final trajectory rather than primarily the QA results.

2 comments
AuthorsOpenReview ↗

TTC Sensitivity and Semantic-Stage Ablations

We thank Reviewer zpAd for the constructive comments. We report additional TTC sensitivity tests and semantic-stage ablations to clarify the role of radar in final planning.

W1/Q1: TTC Reliability

We agree that Doppler radial velocity does not explicitly recover tangential motion. To assess its effect on TTC-based selection, we replace matched radar-cluster velocities with annotated GT planar velocities, retaining radar velocities for unmatched clusters. Candidate trajectories and raw scores remain fixed. Motion types follow annotated GT future motion. TTC MAE averages absolute differences only when both estimates are finite within 3 s. Selection Change measures the percentage of frames selecting a different candidate trajectory. L2 and collision follow the reported open-loop protocol.

Motion TypeTTC MAE (s) ↓Selection Change (%)Radar L2 (m) ↓GT-Vel L2 (m) ↓Radar Coll. (%) ↓GT-Vel Coll. (%) ↓
Longitudinal0.2661.4220.2620.2630.2370.237
Crossing0.3132.1740.3260.3241.4491.449
Turning0.2043.1250.2080.2080.0000.000
Overall0.2721.3270.2590.2590.1750.175

Crossing objects show the highest finite-TTC MAE of 0.313 s, yet only 1.327% of valid frames change candidate selection. Overall L2 and collision remain 0.259 m and 0.175% at the reported precision under both velocity settings. This indicates limited sensitivity for the evaluated candidates, not accurate recovery of tangential motion or true collision-time prediction.

W2/Q2: Contribution to Final Planning

We compare the final RCVLA-Phys results under different semantic-stage variants: without radar–language pretraining, camera-only, and BEV-Planner under 6 s setting.

Semantic-Stage VariantL2 (m) ↓Collision (%) ↓
w/o RLP0.2870.422
w/o 4D Radar0.2620.329
BEV-Planner0.4290.298
Full RCVLA-Sem0.2590.175

Full RCVLA-Sem yields the best final L2 and collision results. Removing radar–language pretraining increases these metrics to 0.287 m and 0.422%, while BEV-Planner yields 0.429 m and 0.298%. These comparisons support the value of radar-informed semantic outputs, although reference trajectories and structured object/occupancy features are not isolated separately. Jointly removing radar-cluster conditioning and TTC calibration from Phys further increases L2 to 0.339 m and collision to 0.391%. These results and methodological limitations are clarified in the revision.

Dear Reviewer zpAd,

We sincerely thank you for your valuable feedback and constructive suggestions. We have carefully addressed your concerns and revised the manuscript accordingly, with all changes highlighted in blue.

We hope our responses adequately address your concerns and welcome any further feedback.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.