Learning Vehicle-centric Risk Representation for Driving Hazard Identification

Identifying hazards that threaten the ego vehicle is a critical capability for intelligent driving systems: it enables timely warnings to drivers and the screening of safety-critical samples from massive driving data. In this paper, we study driving hazard identification: given an ego-view image, determine whether the scene contains hazards relevant to the ego vehicle. This task resists supervised recognition, since hazardous events are rare, diverse, and unpredictable, arising from open-ended combinations of traffic participants, spatial layouts, and behaviors. General-purpose VLMs can query hazards directly with their open-world reasoning; yet without an understanding of which factors actually threaten the ego vehicle, their judgments are easily misled by safety-irrelevant content. We propose RiskRep, which addresses both limitations with a learned, vehicle-centric risk representation. Instead of asking a VLM to judge danger from raw pixels, RiskRep first constructs a vehicle-centric graph (VCG): each traffic participant becomes a node carrying its category attribute, while its edge to the ego vehicle encodes the relative position, heading, and motion. RiskRep then compresses this graph into a compact representation through shared encoding and attention-based aggregation, and trains a risk scoring function on top of it with scene-level hazard supervision. To locate hazard sources, we use the learned scoring function to search for a risk-relevant sub-VCG via counterfactual node removal, and render it as structured hazard cues that guide a frozen VLM toward the final hazard identification. Because RiskRep captures general ego-relative relations rather than specific hazards, our method can generalize to hazard combinations unseen in training. We build a hazard identification benchmark from DrivingDojo and BDD100K, and comprehensive experiments show that RiskRep consistently improves accuracy and F1-scores across VLMs of various scales.