acceptodds
Withdrawn from ICLR 2027

Learning Vehicle-centric Risk Representation for Driving Hazard Identification

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Identifying hazards that threaten the ego vehicle is a critical capability for intelligent driving systems: it enables timely warnings to drivers and the screening of safety-critical samples from massive driving data. In this paper, we study driving hazard identification: given an ego-view image, determine whether the scene contains hazards relevant to the ego vehicle. This task resists supervised recognition, since hazardous events are rare, diverse, and unpredictable, arising from open-ended combinations of traffic participants, spatial layouts, and behaviors. General-purpose VLMs can query hazards directly with their open-world reasoning; yet without an understanding of which factors actually threaten the ego vehicle, their judgments are easily misled by safety-irrelevant content. We propose RiskRep, which addresses both limitations with a learned, vehicle-centric risk representation. Instead of asking a VLM to judge danger from raw pixels, RiskRep first constructs a vehicle-centric graph (VCG): each traffic participant becomes a node carrying its category attribute, while its edge to the ego vehicle encodes the relative position, heading, and motion. RiskRep then compresses this graph into a compact representation through shared encoding and attention-based aggregation, and trains a risk scoring function on top of it with scene-level hazard supervision. To locate hazard sources, we use the learned scoring function to search for a risk-relevant sub-VCG via counterfactual node removal, and render it as structured hazard cues that guide a frozen VLM toward the final hazard identification. Because RiskRep captures general ego-relative relations rather than specific hazards, our method can generalize to hazard combinations unseen in training. We build a hazard identification benchmark from DrivingDojo and BDD100K, and comprehensive experiments show that RiskRep consistently improves accuracy and F1-scores across VLMs of various scales.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Autonmous Driving, risk representation, vision language

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_5njiOpenReview ↗
Rating
1
Confidence
4
Informativeness
3
Presentation
3
Soundness
1
Read review
Paper summary

This paper presents a method to help VLMs recognize driving hazards that actually matter to the ego vehicle. The authors argue that a normal VLM may look at a driving image and notice many objects, but it may not know which objects are relevant to the car's safety. To tackle this, they introduce RISKREP, a framework which represents traffic participants and their ego-relative spatial and motion relations as a vehicle-centric graph, learns a compact risk representation, and extracts risk-relevant subgraphs to guide a frozen VLM.

Critical strengths
  • The reviewer really likes the problem statement of the paper. The central observation that driving risk depends less on object identity alone and more on ego-relative position, motion, and configuration is compelling and well aligned with the task. This gives the paper a meaningful problem formulation rather than being another generic VLM prompting paper.
  • The other thing is based on the reviewer's own experience, the paper’s vehicle-centric structured representation is a sensible inductive bias. The method injects domain structure that is highly relevant to driving safety, rather than relying entirely on a VLM to discover these relations from raw pixels.
Critical weaknesses
  • The paper lacks comparisons against strong cascade-style baselines that closely match the proposed pipeline. RISKREP itself is a cascade: extract traffic participants → construct structured scene information → select/filter potentially risky objects → provide these cues to a VLM for final prediction. However, the main baselines are direct VLM inference, VLM fine-tuning, DriveSafe, and SGG-Intent. None cleanly tests whether the gains come from the proposed learned risk representation versus simply giving the VLM structured, filtered intermediate information. The authors should compare baselines like [1]. This is extremely critical, IMO.
  • The paper says the benchmark is built from 20K scenarios sampled from DrivingDojo and BDD100K, labels a scene risky if a collision or near-collision occurs within three seconds, and uses an 80/20 random split. But it does not provide enough detail to reliably reconstruct the benchmark.

[1] iFinder: Structured Zero-Shot Vision-Based LLM Grounding for Dash-Cam Video Reasoning, NeurIPS 2025

Questions for authors
  • Will you be releasing the dataset?
Reviewer_MZZXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper shows binary driving hazard identification from single ego-view image. RiskRep first use small VLM for extract traffic participants with category, size, heading, etc., and build star-shaped vehicle centric graph. Then authors maps the graph to scene vector, which is trained with cross-entropy on scene labels. Then select top-K sub-graph, and it is rendered as text cues and appended to prompt of frozen VLM to make final decision. The evaluation shows that it is better than direct querying, fine-tuning, DriveSafe and SGG-Intent on three Qwen backbones.

Critical strengths
  • Making risk cue explicit and ego-relative seems reasonable.
Critical weaknesses
  • Sec 5.1 says that the scene is labeled risky if it has "a collision or near-collision within the next three seconds". But DrivingDojo and BDD100K both do not provide such annotation, and paper does not show any additional information of who, how, or what made them. How is "near-collision" defined?
  • VCG edges encode "motion" (Sec. 4.2), but input is one frame, and attributes are coming from 2B VLM whose extraction accuracy is never measured.
  • Attention pooling over set and leave-one-out attribution are standard techniques. Pairwise interaction I_ij (Eq. 13) is computed, but it is not used in selection (Eq. 14). Results are from one split without seeds or confidence intervals, which make me concerned about the novelty.
Questions for authors
  • the ground-truth labels have no described source, so nobody can check the benchmark. The generalization and localization claims are untested, right?
  • Did DriveSafe and SGG-Intent have any access to training labels?
Reviewer_ZhPwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Binary hazard call on one ego-view image (collision or near-collision within 3 seconds). A small VLM builds a star graph around the ego vehicle, a set encoder and attention pool learn a risk score, leave-one-out keeps the high-scoring nodes, and a frozen VLM makes the final call from those text cues. On 20k DrivingDojo/BDD frames, RISKREP beats direct VLM, fine-tuning, DriveSafe, and SGG-Intent on accuracy and F1.

Critical strengths

Table 3 is the result that matters. On the 35B model, the raw VLM is at 66.81 F1, the full graph at 80.90, a size-matched random subgraph at 73.65, and the risk-ranked subgraph at 83.79 F1 / 87.15 accuracy. Filtering is better than a random cut or than dumping every agent, and precision–recall is less one-sided than fine-tuning.

Critical weaknesses

-- Most of the gain is the structured agent list, not the learned risk ranker. Full graph 80.90 F1 vs risk-ranked 83.79. DriveSafe and SGG-Intent are weak controls (reimplemented with “minimal changes,” no cue examples).

-- The generalization sentence is not tested: the split is random 8:2, not a held-out hazard type. “Near-collision” is undefined, the A.1 prompt is a broader task than the 3-second label, motion comes from a 2B VLM on one image with no accuracy check, and the pairwise term (Eq. 13) is never used in the top-K selection (Eq. 14). K is missing. The positive rate looks to be about a third, so this is not the rare-event setting from the intro.

Questions for authors

-- Define near-collision and give the positive rate per split and dataset. Does the gain hold if a hazard type is held out?

-- Show the cue template and K, and say whether Eq. 13 is used. Add thresholded R(G) with no VLM to Table 2.

--What is the 2B extractor’s accuracy on category, position, and motion?

Reviewer_3KaROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses hazard identification from a single ego-view image. As I understand it, the authors' premise is that general-purpose VLMs don't know which scene elements actually matter to the ego vehicle, so they flag scenes as risky. They propose to use a small VLM to turn the image into an ego-centric graph of traffic participants, then train a risk scorer on it to select the top-K most risk-contributing participants via a removal mechanism, and pass these as text cues to a VLM for the final decision. On 20k frames from DrivingDojo and BDD100K, the paper reports the best Acc and F1 across different backbones, compared with zero-shot prompting, finetuning, drivesafe and SGG-intent.

Critical strengths

S1. It presents a motivated problem and a sensible design. I believe that screening safety-critical frames from large driving logs is practically valuable. Getting a trainable model with ego-relative risk capability from a foundation VLM is a reasonable alternative to full VLM finetuning or training from scratch. Therefore, if the gains held up under more rigorous evaluation settings, it'd be a useful recipe.

S2. This shows in ablation that cue selection is important. The full / random /risk-relevant sub-VCG comparison (Table 3) is the right kind of experiment. The finding that the full VCG underperforms the filtered sub-VCG is interesting to me (probably the community).

Critical weaknesses

W1. Benchmark, labels and inputs are not fully described.

  • No established benchmark is used for comparison (e.g, DRAMA, Rank2Tell, DoTA, RiskBench).
  • It is unclear to me how 3-s collision/near-collision labels were obtained (BDD100K does not seem to have such annotations).
  • Node motion and heading are inferred from a single frame by a 2B VLM whose accuracy is not validated.
  • The split is a random 8:2 over frames, so the generalisation to hazard combinations is somehow untested?

W2. Although it seems to have large reported gains, the evaluation doesn't really show what the VLM adds.

  • The fine-tuning baseline is not described. Missing controls include heuristic cue selection (proximity, TTC, attention weights), calibrated or few-shot prompting.
  • The trained risk head R(G) alone reaches 76.38 F1 / 81.34 Acc (Table 4). It outperforms full RiskRep on Qwen3.5-9B (74.64 / 78.53), but it is not reported as a baseline in Table 2 (any reason?). Several entries don't match with their own precision: SGG-Intent 9B with 61.94% across two backbones(?).

W3. Novelty is somehow incremental relative to ego-centric scene-graph risk assessment (Yu Shih-Yuan et al., 2022) and object-removal risk-object identification (C. Li et al., 2020), none of which is discussed. Another work studied a related problem (Korawat, et al).

Yu Shih-Yuan, et al. "Scene-graph augmented data-driven risk assessment of autonomous vehicle decisions." IEEE transactions on intelligent transportation systems 2021.

C. Li, et al. "Who Make Drivers Stop? Towards Driver-centric Risk Assessment: Risk Object Identification via Causal Inference," IROS 2020

Charoenpitaks, Korawat, et al. "Exploring the potential of multi-modal ai for driving hazard prediction." IEEE Transactions on Intelligent Vehicles 2024.

Minor weakness: A number of errors in the manuscript (e.g., Noted that, an vehicle-centric, we randomly sampling, are exclude in this experiments). Please do proofreading carefully.

Questions for authors
  1. The trained risk head R alone reaches 76.38 F1 / 81.34 Acc (Table 4), which outperforms full RiskRep on Qwen3.5-9B (74.64 / 78.53). However, it is not reported as a baseline in Table 2. --> Were all the experiments in Table 2, 3, 4, 5 reported on the same evaluation set? and please explain the inconsistent entries in Tables 1, 2, 4.

  2. How were labels produced for each dataset ?

  3. What are K, the cue template and the encoder details? Is I_ij used anywhere?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.