acceptodds
Under review as a conference paper at ICLR 2027

RADIUS-DRIVE: A RISK-AWARE DIAGNOSTIC BENCHMARK IN SAFETY-CRITICAL AUTONOMOUS DRIVING

Abstract

Autonomous driving (AD) is a safety-critical domain where correct final actions may still be unsupported by reliable risk-aware reasoning. We introduce RADIUS-Drive, a diagnostic benchmark for evaluating safety-critical driving beyond outcome correctness. In particular, a model may produce a correct driving recommendation while failing to recognize the risk trigger, localize the dominant risk element, or correctly identify the factors that should govern the decision, resulting in an outcome-correct yet weakly supported decision. To support auditable evaluation under long-tail conditions, we construct a scalable scenario generation pipeline with controllable reference-based injection and reference-free generation, together with structured supervision for risk perception and decision-relevant factors. We further propose a SAR diagnostic protocol (Safety Awareness Reasoning), where S-Pass, A-Pass, and R-Pass quantify stage-wise safety, risk awareness, and decision reasoning. Complementing these stage-wise metrics, G-Pass provides a graded assessment of cross-phase decision grounding, measuring how well observable risk-aware judgments jointly support the final decision. Benchmarking 15 VLMs and four representative VLM-based AD agents reveals substantial headroom, with G-Pass ranging from to . The results further expose pronounced stage-wise disparities; for example, GPT-5.1 achieves S-Pass but only R-Pass, while the best G-Pass remains . These findings show that strong action-level safety or isolated stage performance does not necessarily translate into well-grounded safety-critical decisions, motivating diagnostic evaluation beyond final-outcome correctness.

Primary area:
datasets and benchmarks
Keywords:
Autonomous Driving, Vision-Language Models, Risk-Aware Reasoning Evaluation, Diagnostic Benchmark, Long-Tail Scenarios

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_DdLfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces RADIUS-Drive, a benchmark for evaluating vision-language models in safety-critical driving scenarios. Its main argument is that a model may produce the correct driving action without actually recognizing or understanding the underlying risk. To address this issue, the authors divide the evaluation into three stages—safety, risk awareness, and decision reasoning—and evaluate 15 general-purpose VLMs and four language-model-based driving agents on 550 image-based scenarios.

Critical strengths
  1. The problem of obtaining the correct answer for the wrong reason is important, especially in safety-critical settings.
  2. The stage-wise evaluation and broad model coverage help identify whether failures arise from risk recognition or decision-making.
Critical weaknesses
  1. Although RADIUS-Drive is positioned as an autonomous driving benchmark, it mainly evaluates VLM reasoning over single images. It excludes temporal and multimodal inputs and relies on a simplified 2D simulator, creating a substantial gap between its broad AD framing and what it actually measures.
  2. The benchmark only checks whether a model’s explanation matches human annotations. It does not establish that the stated risk factors actually caused the decision, and no counterfactual evaluation is provided.
Questions for authors
  1. How can the safety scores be considered reliable when models predict control-related quantities from a single image without depth, motion, or ego-state information?
  2. What evidence shows that the stated risk factors actually drive the model’s decision rather than serving as post-hoc explanations?
3 comments
AuthorsOpenReview ↗

Response to Reviewer DdLf (Part 1/2)

W1: Temporal Dynamics and Evaluation Scope

A: We thank the reviewer for their attention to the evaluation scope of the benchmark. We would like to clarify that our work does not aim to establish a comprehensive benchmark covering all autonomous driving capabilities, but rather focuses on diagnosing risk understanding and decision grounding in safety-critical scenarios.

In fact, numerous studies published at major conferences have investigated related tasks using static driving images. For example, Talk2Car[1] studies visual object grounding in driving scenes, CODA[2] focuses on long-tail anomaly perception, CODA-LM[3] further evaluates long-tail scene understanding and driving recommendations, while DriveQA[4] examines traffic-rule understanding, right-of-way judgment, and driving-related reasoning based on static visual inputs. These studies demonstrate that single-frame visual perception, risk recognition, and decision-relevant reasoning remain meaningful evaluation targets even without temporal dynamics.

RADIUS-Drive specifically targets these diagnostic capabilities in static scenarios, rather than comprehensive temporal driving decision evaluation. Although existing studies can evaluate the correctness of visual understanding or driving recommendations, they still struggle to distinguish whether a model has genuinely identified the source of risk and whether its final decision is supported by corresponding risk evidence.

Similarly, the lightweight 2D simulator in our work is used only to assess immediate collisions and high-dynamic hazards during the Safety phase, rather than to simulate full vehicle dynamics or long-term traffic interactions. Its design prioritizes determinism, computational efficiency, and reproducibility while maintaining a replaceable simulation interface. As explicitly stated in Appendix C.3, the simulator serves as a reference evaluation backend rather than a high-fidelity vehicle dynamics environment. We will open-source the evaluation pipeline, allowing users to replace the reference simulator with their own simulators.

Therefore, we consider RADIUS-Drive complementary to temporal, multimodal, and closed-loop driving benchmarks, as they evaluate different capability dimensions. The contribution of our work lies in addressing the limitations of conventional outcome-oriented evaluation by introducing diagnosis of risk understanding and cross-phase decision grounding, rather than replacing comprehensive evaluation of complete autonomous driving systems. We will further clarify its research positioning as an image-based diagnostic benchmark for safety-critical driving, along with its scope and limitations, in the revised manuscript.

[1] Talk2Car: Taking Control of Your Self-Driving Car, Deruyttere et al., EMNLP 2019

[2] CODA: A Real-World Road Corner Case Dataset for Object Detection in Autonomous Driving, Li et al., ECCV 2022

[3] Automated Evaluation of Large Vision-Language Models on Self-Driving Corner Cases, Chen et al., WACV 2025

[4] Passing the Driving Knowledge Test, Wei et al., ICCV 2025

W2&Q2: Counterfactual Validation of Risk-Driven Decisions

A: We thank the reviewer for this question. We further designed a counterfactual perturbation experiment to examine the actual influence of intermediate risk judgments on final driving decisions.

Specifically, we adopt a two-stage evaluation procedure. First, we obtain the model's structured risk judgments and decision-relevant factors for a given scene. Subsequently, while keeping the original scene image and all other inputs unchanged, we introduce controlled perturbations to the model's intermediate risk-factor judgments and regenerate the final driving strategy. By comparing final-decision accuracy before and after perturbation, we measure the extent to which the model relies on intermediate risk judgments.

The experimental results show that perturbing intermediate risk judgments leads to decreases of 20.83, 25.00, and 22.50 percentage points in final-decision accuracy for GPT-4o, Gemini-2.5-Flash, and GPT-5.1, respectively. These results indicate that changes in intermediate risk factors have a substantial influence on the models' final decisions, rather than merely serving as post-hoc explanations independent of the decision-making process. This counterfactual experiment further supports the necessity of jointly diagnosing risk recognition, decision-relevant factors, and final driving strategies in RADIUS-Drive.

We will include the above experimental setup and results in the revised manuscript.

ModelΔ Decision Accuracy (pp) ↓
GPT-4o-20.83
Gemini-2.5-Flash-25.00
GPT-5.1-22.50
AuthorsOpenReview ↗

Response to Reviewer DdLf (Part 2/2)

Q1: Reference Scene States for Safety Evaluation

A: We thank the reviewer for their attention to the reliability of Safety scoring. We would like to clarify that we construct simulator-ready scene-state annotations containing spatial and kinematic information for each sample.

Specifically, Appendix C.3 of the original manuscript already explains that each sample includes a coarse 2D road representation and kinematic state, including road geometry, road boundaries, and the categories, positions, and velocities of surrounding traffic participants. The longitudinal axis is aligned with the ego vehicle's forward direction, while the lateral axis represents cross-lane motion. Appendix D.4.1 further specifies that we construct a simulator-ready scene state for each scenario, including the ego state, road geometry, and the positions, dimensions, and plausible velocities of surrounding traffic participants. We also annotate actions in the discrete control space according to whether they result in collisions, trigger high-dynamic hazards, or remain safe. This information is stored in accompanying data files, such as dataX_aligned.json, for use by the deterministic simulator, rather than requiring the model to directly predict these physical states from the image alone.

Therefore, the Safety evaluation follows a clear procedure: the model predicts a pre-decision action based on the image, while the simulator independently evaluates that action through short-horizon kinematic simulation and collision detection using the corresponding annotated physical scene state. This avoids directly relying on potentially inaccurate spatial or motion estimates produced by the evaluated model itself and ensures that different models are evaluated under identical scene states and simulation conditions.

We will further clarify the specific contents of these physical-state annotations in the revised manuscript.

AuthorsOpenReview ↗

Follow-up on Our Response

Dear Reviewer DdLf,

Thank you again for your valuable feedback. We would like to follow up on our previous response.

To address your main concerns, we have further clarified RADIUS-Drive's research positioning and evaluation scope, emphasizing that our work aims to diagnose risk understanding and decision grounding in static safety-critical scenarios, rather than replace comprehensive temporal or closed-loop driving evaluations. Meanwhile, we have conducted additional counterfactual perturbation experiments, demonstrating the significant sensitivity of models' final decisions to intermediate risk judgments. In addition, we have further clarified that the physical states used for Safety scoring are derived from independent scene annotations, rather than quantities predicted by the models from a single image, and provided a detailed explanation of the evaluation mechanism and applicable scope of our short-horizon simulation.

We hope these clarifications help resolve potential misunderstandings and adequately address your main concerns. Furthermore, the additional experimental evidence provides further support for the core contributions and evaluation value of our work. If you have any remaining questions, we would be more than happy to provide further clarification and discussion. If you believe that these clarifications and additional experiments have sufficiently addressed your previous concerns, we sincerely ask that you reconsider the research contributions of our work in light of the more comprehensive evidence and consider revising your overall assessment and score accordingly.

Thank you again for taking the time to review our work. We look forward to hearing your further feedback.

Best regards,

The Authors

Reviewer_NPYGOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

RADIUS-Drive evaluates whether driving recommendations are accompanied by correct risk recognition and decision-relevant judgments. It introduces 550 image-based scenarios and the Safety–Awareness–Reasoning protocol, with experiments covering 15 VLMs and four driving agents.

Critical strengths

The paper addresses a useful evaluation gap: a correct recommendation can conceal errors in risk understanding. The structured risk and factor annotations make these errors visible and provide more diagnostic detail than final-answer accuracy alone.

Critical weaknesses

A-Pass and R-Pass are evaluated on each model’s own collision-free subset. Consequently, models are scored on different samples, which complicates comparisons in Table 2. Reporting gate coverage and results on a common sample set would help determine whether the rankings reflect capability differences or sample selection.

I could not reconstruct G-Pass unambiguously from the current description. The normalization of (PiP_i) and (QiQ_i) needs explicit formulas, and Appendices C.6 and C.7 refer to each other without clearly assigning downstream scores to samples that fail the gate. A worked example would help resolve this.

The safety interpretation remains limited by synthetic single-image scenarios and a short rollout using coarse motion annotations. The benchmark is useful for structured diagnosis, but its relationship to closed-loop driving reliability needs further validation. Always-brake and text-only controls would also help test whether performance depends on scene-specific understanding.

I lean toward weak acceptance. The benchmark offers a useful way to expose errors that final-answer evaluation misses, and the diagnostic examples and validation checks give the contribution substance. The scoring ambiguities and gating effects remain important issues for the authors to address.

Questions for authors

How do A-Pass and R-Pass rankings change when evaluated on a common sample set? Please also report each model’s collision-free gate coverage.

How are image-derived positions and velocities validated, and what physical scale do the simulator units represent? How sensitive are Safety scores to these annotations?

How do always-brake, fixed conservative-response, and text-only baselines perform? These controls would help establish whether the benchmark rewards scene-specific risk understanding.

What are the exact normalization rules for (PiP_i) and (QiQ_i), and how are downstream G-Pass components assigned when a sample fails the collision-free gate

3 comments
AuthorsOpenReview ↗

Response to Reviewer NPYG (Part 1/3)

W1&Q1: Gate Coverage and Common-Subset Evaluation

A: We thank the reviewer for this question. We agree that reporting Gate Coverage for each model and conducting supplementary comparisons on a common sample set would help distinguish model capability differences from sample selection effects.

First, the original Figure 11 already provides a fine-grained decomposition of Safety Errors for 15 general-purpose VLMs, where Collision corresponds to the proportion of samples that fail the collision-free gate. Therefore, we can directly extract the collision rate of each model from these statistics and calculate Gate Coverage as its complement:

Coveragem=1−Ncoll,m/NCoverage_m = 1 - N_{coll,m}/N

We will further organize the Gate Coverage of all models into a numerical table and supplement the corresponding statistics for AD-specific agents from the original manuscript, making the gating coverage across different models more transparent.

Second, to eliminate the influence of model-specific sample selection on A-Pass and R-Pass rankings, we further construct a common sample set consisting of samples that pass the collision-free gate for all models, and recompute both metrics on this fixed set. The results show that model rankings on the common sample set remain highly consistent with those in the original Table 2: except for GLM-4.6V and Gemini-3-pro, all other models retain their original rankings. We further measure ranking consistency between the two evaluation settings using Spearman's ρ and Kendall's τ, and the results also support the stability of the model comparison conclusions.

These results indicate that although collision-free gate coverage varies across models, the resulting sample selection effects do not substantially alter the overall model capability rankings. We will supplement the revised manuscript with Gate Coverage for each model, the size of the common evaluation set, and ranking consistency statistics to further improve the transparency and comparability of the evaluation results.

MetricSpearman's ρ ↑Kendall's τ-b ↑
A-Pass0.9860.942
R-Pass0.9420.863


W2&Q4: G-Pass Computation and Gate Handling

A: We thank the reviewer for pointing out the lack of clarity in the implementation details of G-Pass.

Specifically, (PiP_i) measures vehicle safety performance in the pre-decision phase. We divide it into three levels: collision, collision-free but triggering a high-dynamic hazard, and fully safe, assigning normalized scores of 0, 0.5, and 1, respectively. (QiQ_i) directly adopts (qidecq_i^{dec}) from Eq. (6) of the original manuscript, without introducing any additional scoring scale.

Regarding the collision-free gate, samples that fail the gate due to a collision are excluded from the conditional evaluation of subsequent Awareness and Reasoning, since the collision has already occurred in the pre-decision phase, and subsequent risk judgments and decision reasoning no longer operate under a valid collision-free driving condition. All downstream components are assigned zero, resulting in zero contribution to G-Pass. In contrast, collision-free samples that trigger a high-dynamic hazard still pass the gate and participate normally in downstream evaluation, with only their pre-decision performance score (PiP_i) set to 0.5.

To make the computation more intuitive, we further provide a worked example. For a sample that passes the gate, suppose (Pi=Qi=Li=Ci=1P_i=Q_i=L_i=C_i=1), (Ei=0E_i=0), (Fi=Vi=2/3F_i=V_i=2/3), and (Γi=1/2\Gamma_i=1/2). According to Eq. (10) of the original manuscript, the normalized G-Pass score for this sample is:

gi4=1+1+0+2/934≈0.6514.\frac{g_i}{4}=\frac{1+1+0+\sqrt[3]{2/9}}{4}\approx0.6514.

This illustrates that even when the model produces a correct final strategy, incorrect risk localization and incomplete effective-factor judgments still reduce its overall score.

We will explicitly supplement the revised manuscript with the above normalization formulas, gate-handling rules, and a complete worked example to eliminate the ambiguity caused by cross-references between Appendices C.6 and C.7. In addition, we will open-source the complete evaluation pipeline, including the scoring implementation, configuration files, prompt templates, and reference simulator, to ensure that G-Pass and the entire SAR evaluation protocol can be independently reproduced and further validated.

AuthorsOpenReview ↗

Response to Reviewer NPYG (Part 2/3)

Q2: Physical-State Annotations and Safety Evaluation Reliability

A: We thank the reviewer for their attention to physical-state annotations and the reliability of Safety evaluation. We would like to further clarify three aspects.

First, the physical-state information in each scenario undergoes human annotation and verification. We convert each image into a coarse 2D scene representation, manually annotating road geometry, traffic participants' categories, positions, dimensions, and plausible planar velocities. Human quality checks are then conducted to verify consistency with the original scene. Appendix D.7 already describes our human annotation team and quality-control procedures.

Second, the simulator adopts a unified coordinate scale with explicit physical meaning: one simulation unit corresponds to 0.02 m. Positions are represented at this scale, while velocities and accelerations are expressed in unit/s and unit/s², respectively. Under this scale, the default collision-boundary expansion margins of 8, 15, and 35 units correspond to 0.16 m, 0.30 m, and 0.70 m, respectively. This ensures that the geometric representations and collision boundaries in the simulator have well-defined physical scales.

Finally, Safety scoring relies on deterministic short-horizon simulation based on classical kinematic equations, rather than a learned evaluator. Given the initial vehicle state, predicted action, and annotated motion states of surrounding traffic participants, we perform a 3-second rollout using fixed kinematic update equations and determine whether a collision or high-dynamic hazard occurs through geometric boundary checks. All models use the same simulator, physical scale, and fixed parameters, ensuring fully consistent and reproducible Safety judgments under identical input conditions.

These designs establish Safety evaluation on human-verified physical scene representations and standardized deterministic kinematic simulation, rather than subjective judgments or model-dependent evaluation criteria.

AuthorsOpenReview ↗

Response to Reviewer NPYG (Part 3/3)

W3&Q3: Evaluation Scope and Control Baselines

A: We thank the reviewer for this question. First, we would like to further clarify the scope of RADIUS-Drive: our work does not aim to evaluate comprehensive closed-loop driving reliability, but rather focuses on diagnosing risk understanding and decision grounding in static safety-critical scenarios.

Numerous studies published at major conferences have investigated related tasks using static driving images. For example, Talk2Car[1] studies visual object grounding in driving scenes, CODA[2] focuses on long-tail anomaly perception, CODA-LM[3] further evaluates long-tail scene understanding and driving recommendations, while DriveQA[4] examines traffic-rule understanding, right-of-way judgment, and driving-related reasoning based on static visual inputs. These studies demonstrate that single-frame visual perception, risk recognition, and decision-relevant reasoning remain meaningful evaluation targets even without temporal dynamics. We acknowledge the importance of closed-loop driving evaluation, but consider it complementary to the structured risk diagnosis addressed in this work.

In response to the reviewer's suggestion regarding control baselines, we further evaluate two strategies, Always-brake and Fixed conservative-response, using G-Pass as the unified evaluation metric.

The primary objective of these experiments is to examine whether relying solely on conservative behaviors is sufficient to achieve high cross-phase decision grounding scores. Always-brake consistently applies a fixed braking action, while Fixed conservative-response adopts the same conservative driving strategy across all scenarios. The experimental results show that neither fixed strategy achieves a high G-Pass score.

These results are consistent with the design objective of G-Pass. On the one hand, long-tail scenarios in RADIUS-Drive involve diverse passability constraints, and braking or stopping is not appropriate in every scenario. Some scenarios require timely continuation or other appropriate maneuvers. Fixed braking or conservative responses may therefore conflict with the reference decisions in these scenarios, reducing the decision-performance component of G-Pass. On the other hand, G-Pass evaluates not only the final strategy but also jointly considers risk category recognition, dominant-risk localization, and factor-validity judgments. Without scene-specific risk understanding, a model cannot obtain sufficient credit across these diagnostic dimensions, even if it produces safe actions in some scenarios.

These control experiments further demonstrate that RADIUS-Drive does not simply reward conservative driving behaviors, but instead emphasizes consistency between risk understanding and the final decision in specific scenarios. Models need to select appropriate driving strategies according to the actual risk mechanisms, rather than relying solely on always braking or fixed conservative responses to achieve high evaluation scores.

Regarding the Text-only baseline, we consider it not directly comparable under the current benchmark input setting. RADIUS-Drive tasks depend on specific risk elements and spatial relationships in the images. Without images, models would lack sample-specific scene evidence, whereas providing additional textual scene descriptions would introduce extra scene-description or perceptual information, thereby changing the original evaluation input conditions. Therefore, we do not include this setting as a standard baseline for direct comparison with the original multimodal models.

ModelStrategyG-Pass ↑Δ G-Pass
Qwen3-VL-235BAlways-brake0.33-0.15
Qwen3-VL-235BFixed conservative-response0.36-0.12
GPT-5.2Always-brake0.35-0.18
GPT-5.2Fixed conservative-response0.39-0.14

[1] Talk2Car: Taking Control of Your Self-Driving Car, Deruyttere et al., EMNLP 2019

[2] CODA: A Real-World Road Corner Case Dataset for Object Detection in Autonomous Driving, Li et al., ECCV 2022

[3] Automated Evaluation of Large Vision-Language Models on Self-Driving Corner Cases, Chen et al., WACV 2025

[4] Passing the Driving Knowledge Test, Wei et al., ICCV 2025

Reviewer_2DorOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RADIUS-Drive, a diagnostic benchmark for evaluating risk-aware decision-making in autonomous driving. It introduces a Safety–Awareness–Reasoning (SAR) protocol and G-Pass metric to assess whether driving decisions are supported by correct risk perception and reasoning. The benchmark contains 550 scenarios generated through controllable risk injection and scene generation. Experiments on 15 VLMs and four autonomous driving agents reveal significant gaps between action correctness and risk-aware reasoning.

Critical strengths
  1. The benchmark addresses an important problem by evaluating risk-aware reasoning beyond final driving decisions.
  2. The SAR protocol provides a structured evaluation of safety, risk awareness, and decision reasoning.
  3. Extensive comparisons across VLMs and autonomous driving agents demonstrate differences in their safety-related capabilities.
Critical weaknesses
  1. The benchmark contains only 550 image-based scenarios, mostly generated or edited. It remains unclear whether these scenarios adequately represent complex real-world driving conditions.
  2. The weighting of different components in A-Pass, R-Pass, and G-Pass is largely heuristic. The paper lacks sufficient justification that these scores reliably reflect actual driving safety or decision quality.
  3. Autonomous driving requires reasoning about dynamic interactions over time. However, the benchmark primarily uses static images and short-horizon simulated actions, which may not capture long-term decision consequences.
  4. The benchmark evaluates structured risk judgments rather than the model's actual reasoning process. Correct intermediate answers do not necessarily indicate that the model used this information to make its final decision.
Questions for authors
  1. Can the authors extend the evaluation to temporal scenarios or closed-loop driving simulations?
  2. How can the proposed metrics distinguish genuine risk-aware reasoning from models that produce plausible intermediate answers without using them for decision-making?
5 comments
AuthorsOpenReview ↗

Response to Reviewer 2Dor (Part 1/4)

W1: Scenario Coverage and Real-World Representativeness

A: We thank the reviewer for their attention to dataset scale and real-world scenario representativeness. We would like to further clarify three aspects: evaluation stability, long-tail coverage, and scenario realism.

First, 550 samples are already sufficient to support stable model diagnosis and comparison. We conducted stability experiments across different dataset sizes in Appendix E.2 (Table 8). The results show that when the sample size increases from 300 to 550, changes in all four core metrics (S-Pass, A-Pass, R-Pass, and G-Pass) remain within 0.02, while model rankings remain completely unchanged (Spearman ρ = 1.0). This indicates that the evaluation results at the current scale already exhibit good statistical stability, and that further increasing the sample size would have limited impact on the existing model comparison conclusions.

Second, we place greater emphasis on systematic coverage of long-tail risk mechanisms rather than simply pursuing a larger number of scenarios. Real-world driving scenarios and their combinations cannot be exhaustively enumerated by any finite dataset. To this end, RADIUS-Drive establishes a three-level safety-critical risk taxonomy, in which 500 long-tail scenarios cover three passability regimes and 15 risk mechanisms, with at least 10 samples in each Level-2 category. Figure 3 and Appendix B (Figure 6) further illustrate the scenario distribution and the expansion of risk-element types relative to existing long-tail datasets. In addition, we adopt a taxonomy-driven controllable generation pipeline that combines risk injection into real images with reference-free generation, allowing us to continuously incorporate new risk elements and their specific instantiations without modifying the existing evaluation framework.

Finally, to assess the realism of generated and edited scenarios, we further designed a blind comparative evaluation against real long-tail scenarios. Specifically, we randomly mixed 100 real long-tail scenarios with 100 RADIUS-Drive-constructed scenarios and asked five independent reviewers to evaluate them. The results show that the constructed scenarios are close to real scenarios in terms of visual realism (4.82 vs. 4.59), risk-mechanism identification agreement (94.0% vs. 96.0%), and inter-rater agreement (κ = 0.89 vs. 0.86). These results indicate that our generation pipeline can effectively preserve the key risk semantics of real long-tail scenarios while maintaining high visual realism and supporting human diagnosis with good agreement.

MetricReal Long-TailRADIUS-Drive
Visual Realism (1–5) ↑4.824.59
Risk-Mechanism Agreement ↑94.00%96.00%
Inter-Rater Agreement (Fleiss' κ) ↑0.890.86


AuthorsOpenReview ↗

Response to Reviewer 2Dor (Part 2/4)

W2: Metric Weighting and Evaluation Validity

A: We thank the reviewer for their attention to the metric weighting design and evaluation validity. We would like to further clarify that the primary goal of RADIUS-Drive is not to approximate real-world driving safety through a single weighted score, but to provide interpretable, structured diagnosis of model safety performance, risk awareness, and decision grounding. Therefore, we adopt a hierarchical scoring design involving component normalization, within-stage aggregation, and cross-dimensional coupling.

First, all basic scoring components are normalized to [0, 1] to prevent differences in numerical scales across evaluation items from affecting the aggregated results. For A-Pass (Eq. 3–4), we assign weights according to each component's relevance to driving decisions: risk category recognition and passability-constraint judgment directly reflect the risk mechanism and its constraints on driving behavior, whereas long-tail trigger recognition and dominant-risk localization primarily provide the foundation for risk identification. For R-Pass (Eq. 6–8), we adopt an equal-weight average of factor-validity judgment and final-decision correctness, preventing overly high scores based solely on a correct final decision or isolated factor judgments. Multiple factor judgments are first averaged internally to prevent the number of factors from affecting their overall weight.

Second, G-Pass (Eq. 9–11) equally aggregates four high-level diagnostic dimensions and uses geometric means to couple semantically dependent components. For judgments that must jointly hold, such as risk category recognition and dominant-risk localization, or passability-factor recognition and factor-validity judgment, we adopt geometric means and introduce the effective-factor agreement term Γ to prevent high scores in individual components from masking errors in other critical judgments. This design directly serves the central motivation of our work: identifying Pseudo-Correctness, where the final decision is correct but lacks sufficient support from risk awareness and reasoning, rather than simply summing the accuracy of independent components.

Overall, our scoring framework has relatively low dependence on manually assigned weights. Through component normalization, reasonable within-stage aggregation, and equal weighting across diagnostic dimensions, we aim to ensure fairness and comparability across different evaluation dimensions as much as possible. Meanwhile, all weights and aggregation rules are fully transparent and consistently applied across all models. We will also open-source the complete evaluation pipeline, allowing researchers to flexibly adjust weights and aggregation methods according to specific task requirements, thereby further validating and extending the applicability of RADIUS-Drive.

AuthorsOpenReview ↗

Response to Reviewer 2Dor (Part 3/4)

W3 & Q1: Temporal Dynamics and Long-Horizon Evaluation

A: We thank the reviewer for this question. We agree that these capabilities are essential for complete autonomous driving systems. However, we would like to clarify that risk understanding and decision reasoning in static scenarios are also indispensable foundational capabilities for autonomous driving and constitute an evaluation direction with independent research value.

In fact, numerous studies published at major conferences have investigated related tasks using static driving images. For example, Talk2Car[1] studies visual object grounding in driving scenes, CODA[2] focuses on long-tail anomaly perception, CODA-LM[3] further evaluates long-tail scene understanding and driving recommendations, while DriveQA[4] examines traffic-rule understanding, right-of-way judgment, and driving-related reasoning based on static visual inputs. These studies demonstrate that single-frame visual perception, risk recognition, and decision-relevant reasoning remain meaningful evaluation targets even without temporal dynamics.

RADIUS-Drive specifically targets these diagnostic capabilities in static scenarios, rather than comprehensive temporal driving decision evaluation. Although existing studies can evaluate the correctness of visual understanding or driving recommendations, they still struggle to distinguish whether a model has genuinely identified the source of risk and whether its final decision is supported by corresponding risk evidence. To address this, we introduce the SAR diagnostic protocol, which separately evaluates Safety, Awareness, and Reasoning, and uses G-Pass to measure how observable risk judgments support the final decision, thereby identifying Pseudo-Correctness. The lightweight short-horizon simulation is used only to provide standardized and reproducible conditions for immediate safety evaluation, rather than to infer long-term driving consequences.

Therefore, we consider static risk diagnosis and temporal interaction evaluation to be two complementary research directions. The contribution of RADIUS-Drive lies in addressing the limitations of existing diagnostic evaluation in static safety-critical scenarios. We will further clarify this research positioning and its scope in the revised manuscript.

[1] Talk2Car: Taking Control of Your Self-Driving Car, Deruyttere et al., EMNLP 2019

[2] CODA: A Real-World Road Corner Case Dataset for Object Detection in Autonomous Driving, Li et al., ECCV 2022

[3] Automated Evaluation of Large Vision-Language Models on Self-Driving Corner Cases, Chen et al., WACV 2025

[4] Passing the Driving Knowledge Test, Wei et al., ICCV 2025

Comments on the paper

1 comment

Dear Reviewers,

The rebuttal deadline is approaching. I am leaving this friendly reminder to encourage active discussion. Thank you for your valuable time and engagement!

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.