Drive What Matters: Learning and Forecasting Decision Relevance for End-to-End Autonomous Driving

Recent end-to-end driving methods combine current scene understanding with future scene prediction to support planning. However, predicting how the scene evolves does not explicitly reveal which elements are decision-critical. An actor that is irrelevant to the current decision may become critical under a particular future maneuver. It is therefore essential to model both an actor’s current decision relevance and its future evolution. To this end, we introduce DriveMatter, a framework that learns current decision relevance and forecasts future evolution for two-stage trajectory refinement. To supervise this learning, we introduce complementary geometric and semantic supervision that captures route-conditioned spatial relevance and interaction-driven importance for planning. The learned current relevance first refines a base route and target speed into a candidate plan. Conditioned on this candidate and observation history, a Plan-Conditioned Relevance World Model (PCR-WM) forecasts future relevance to guide a second refinement of route and speed. Together, these stages allow the planner to respond to what matters now and anticipate what will matter as the candidate maneuver unfolds. On the Bench2Drive closed-loop benchmark, DriveMatter consistently improves success rate across distinct expert-data settings. Benefiting from joint modeling of current decision relevance and its future evolution, it outperforms the corresponding baselines by 10.91 and 8.1 percentage points with PDM-Lite demonstrations and LEAD expert data, respectively.