acceptodds
Under review as a conference paper at ICLR 2027

Drive What Matters: Learning and Forecasting Decision Relevance for End-to-End Autonomous Driving

Abstract

Recent end-to-end driving methods combine current scene understanding with future scene prediction to support planning. However, predicting how the scene evolves does not explicitly reveal which elements are decision-critical. An actor that is irrelevant to the current decision may become critical under a particular future maneuver. It is therefore essential to model both an actor’s current decision relevance and its future evolution. To this end, we introduce DriveMatter, a framework that learns current decision relevance and forecasts future evolution for two-stage trajectory refinement. To supervise this learning, we introduce complementary geometric and semantic supervision that captures route-conditioned spatial relevance and interaction-driven importance for planning. The learned current relevance first refines a base route and target speed into a candidate plan. Conditioned on this candidate and observation history, a Plan-Conditioned Relevance World Model (PCR-WM) forecasts future relevance to guide a second refinement of route and speed. Together, these stages allow the planner to respond to what matters now and anticipate what will matter as the candidate maneuver unfolds. On the Bench2Drive closed-loop benchmark, DriveMatter consistently improves success rate across distinct expert-data settings. Benefiting from joint modeling of current decision relevance and its future evolution, it outperforms the corresponding baselines by 10.91 and 8.1 percentage points with PDM-Lite demonstrations and LEAD expert data, respectively.

Primary area:
applications to robotics, autonomy, planning
Keywords:
End-to-End Autonomous Driving, Decision Relevance, World Models, Trajectory Planning, Representation Learning

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_KK6nOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes DriveMatter, an end-to-end autonomous driving framework that explicitly models the decision relevance of surrounding traffic participants. The authors introduce geometric and LLM-generated semantic supervision to identify planning-relevant actors and use the learned representations to refine trajectories and target speeds. A Plan-Conditioned Relevance World Model (PCR-WM) further predicts future relevance conditioned on candidate trajectories, enabling a second planning refinement. Experiments on the Bench2Drive closed-loop benchmark demonstrate improvements across two expert-data settings, achieving success-rate gains of 10.91 and 8.1 percentage points over the corresponding baselines.

Critical strengths
  1. Clear motivation and practical relevance. The paper addresses an important limitation of conventional scene-centric representations by explicitly modeling which traffic participants are relevant to driving decisions and how their relevance evolves over time.

  2. Intuitive and interpretable design. The proposed geometric and semantic supervision provides an explicit representation of decision relevance. The combination of current relevance estimation and future relevance prediction offers a reasonable approach to improving trajectory planning.

  3. Strong closed-loop results and comprehensive experiments. DriveMatter achieves competitive performance on Bench2Drive, with consistent improvements across two different base planners and expert-data settings. The comparisons with image- and occupancy-based world models, together with component ablations, further support the effectiveness of relevance-oriented supervision.

Critical weaknesses
  1. Unclear necessity of two-stage refinement. The framework performs two successive trajectory refinements, but lacks comparisons with simpler single-stage fusion strategies. It remains unclear whether jointly attending to current and future relevance features could achieve comparable performance with a simpler architecture.

  2. Insufficient ablation of the World Model's contribution. The improvement from 75.00% to 78.18% SR involves both PCR-WM and an additional refinement stage. A two-stage refinement baseline without the World Model would help isolate the contribution of future relevance prediction from that of the additional refinement module.

  3. Real-world generalization. The experiments are conducted entirely in the CARLA/Bench2Drive environment, leaving real-world generalization unexplored. Additional real-world evaluations would further strengthen the practical relevance of the work. However, I recognize the substantial effort required for such experiments and do not consider their absence a factor in my rating.

Questions for authors

N/A

7 comments

We sincerely thank the reviewer KK6n for the constructive feedback on the two-stage refinement design, the contribution of the World Model, and real-world generalization. The comments have helped improve the experimental validation and clarify the scope and limitations of DriveMatter. Below, we address all three concerns (W1–W3) point by point. The corresponding additional experiments and revisions have been incorporated into the revised manuscript.

W1: Necessity of Two-stage Refinement

We thank the reviewer for this helpful suggestion. To evaluate the benefit of staged refinement over single-stage fusion, we introduce a Single-stage Joint Fusion baseline. The Joint Fusion baseline and Full DriveMatter use the same current and future relevance information but differ in how they guide planning. The Joint Fusion baseline retains the first-stage current-relevance refinement. However, its WM predicts future relevance conditioned on the trajectory generated by the base planner. The current and predicted future relevance features are then jointly integrated in a single subsequent refinement. In contrast, Full DriveMatter conditions PCR-WM on the candidate plan generated by the first-stage refinement and uses the resulting future relevance to guide the second-stage refinement.

Table 1. Single-stage Joint Fusion vs. Two-stage Refinement

ModelRefinement StrategyDS ↑SR ↑Multi-Ability ↑
Joint FusionSingle-stage88.7076.82%73.07%
Full DriveMatterTwo-stage89.2478.18%74.23%

Two-stage DriveMatter outperforms Joint Fusion by 0.54 DS, 1.36 percentage points in SR, and 1.16 percentage points in Multi-Ability. The improvements are most pronounced in overtaking (+2.22 pp) and merging (+2.50 pp).

The key advantage of staged refinement lies in explicitly coupling current planning with future interaction prediction. Current relevance establishes a candidate maneuver, which then conditions PCR-WM to predict how interaction relevance evolves under that maneuver. The second stage uses these predictions to further refine the route and target speed, allowing future interaction constraints to directly inform the planned action.

W2: Contribution of the World Model

Thank you for this suggestion. We agree that the original comparison between Current-only and the full DriveMatter simultaneously introduces future prediction and a second refinement stage, making it difficult to isolate their respective contributions.

To disentangle these effects, we introduce a Two-stage w/o WM baseline. This variant retains the same two-stage refinement architecture and second-stage planner as the full DriveMatter, but uses current relevance features and fields instead of world-model predictions to guide the second refinement.

Table 2. Ablation of Second-stage Refinement and the World Model

ModelFirst RefinementSecond RefinementWorld ModelDS ↑SR ↑Multi-Ability ↑
Current-only✓✗✗88.1175.00%70.23%
Two-stage w/o WM✓✓✗88.3575.45%70.86%
Full DriveMatter✓✓✓89.2478.18%74.23%

The results separate the contributions of the two components. Adding a second refinement stage improves DS, SR, and Multi-Ability by 0.24, 0.45, and 0.63 percentage points, respectively. Incorporating PCR-WM under the same two-stage architecture provides further gains of 0.89 DS, 2.73 percentage points in SR, and 3.37 percentage points in Multi-Ability.

The results show that the benefit of the second stage comes primarily from the future prediction information provided by PCR-WM, rather than simply adding another refinement operation. By conditioning future relevance prediction on the candidate plan generated by the first stage, PCR-WM enables the planner to anticipate evolving interaction constraints and jointly refine the route and target speed. This establishes the contribution of future relevance prediction beyond the additional refinement operation itself.

W3: Real-world Generalization

We thank the reviewer for highlighting the importance of real-world generalization. Real-world evaluation is indeed an important step toward assessing the practical applicability of end-to-end autonomous driving methods.

All experiments in this work are conducted in the CARLA environment. DriveMatter demonstrates consistent improvements across two configurations, PDM-Lite/TransFuser++ and LEAD/TFv6, covering different planning backbones and expert-data settings. Nevertheless, real-world generalization remains an important direction for further investigation. Future work will focus on real-world driving data and closed-loop vehicle experiments to evaluate Sim-to-Real generalization.

This limitation will be explicitly acknowledged in the revised manuscript under Limitations and Future Work.

We sincerely appreciate the reviewer's understanding and constructive suggestion.

Reviewer_JveEOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DriveMatter, an end-to-end autonomous driving framework that explicitly models which scene elements matter for ego planning. It combines geometric and LLM-generated semantic supervision to learn current decision relevance, and introduces a Plan-Conditioned Relevance World Model (PCR-WM) to predict how relevance evolves under a candidate ego plan. Current and predicted future relevance are used in two stages to refine the planned route and target speed. Experiments on Bench2Drive show improved closed-loop performance over the base planners and several existing methods.

Critical strengths
  • The motivation is clear. Explicitly modeling decision relevance, rather than relying only on general scene representations or future scene prediction, is an interesting direction for autonomous driving.
  • The paper is well-written and well-structured. The combination of geometric and semantic relevance supervision, plan-conditioned future prediction, and two-stage planning refinement is intuitive.
  • The paper reports improvements under both PDM-Lite and LEAD expert settings, and includes comparisons with RGB/occupancy prediction, relevance supervision ablations, and closed-loop interaction examples.
Critical weaknesses
  • A key contribution of this paper is PCR-WM, which predicts future decision relevance conditioned on the candidate ego plan. However, the experiments do not directly isolate whether this conditioning is necessary. Table 4 compares the complete model with a current-relevance-only variant, but this simultaneously introduces future prediction and a second planning refinement stage. Can you compare PCR-WM with and without candidate-plan conditioning, while keeping the remaining architecture and refinement process unchanged. It would also be useful to show how the predicted relevance changes when different feasible candidate plans are provided for the same scene.

  • The semantic supervision is generated by an offline LLM using structured actor information and privileged future interaction cues. The paper reports a 7.96% semantic label error rate from manual review, which is useful. However, the Critical class is particularly important because it influences braking and yielding decisions, and the paper acknowledges that distinguishing Critical from Relevant is challenging. Could the authors provide class-wise annotation quality, especially for Critical actors, and discuss how label errors affect planning?

  • The paper defines decision relevance as depending on ego state and driving intent, and PCR-WM is designed to predict relevance under a candidate maneuver. However, the semantic labels are constructed offline using recorded trajectories and privileged future interaction information. It is unclear how these labels should be interpreted when the candidate plan differs from the recorded maneuver. Does PCR-WM learn relevance specifically for the candidate action, or primarily predict relevance associated with the recorded future?

Questions for authors
  • Section 3.4 introduces Cross-Attention, Local Sampling, Gated Residual, and Slowdown Bias, but their interactions are described only briefly. In particular, the construction of the Critical-based Slowdown Bias and its effect on target-speed logits could be explained more explicitly.
6 comments

We sincerely thank Reviewer JveE for the constructive feedback on candidate-plan conditioning, semantic annotation quality, and the interpretation of offline supervision under different candidate maneuvers. Below, we address all three concerns (W1–W3) and the question (Q1) point by point. We provide additional matched ablations, candidate-plan visualizations, class-wise annotation analyses, and clarifications of the planning refinement components. All corresponding revisions have been incorporated into the revised manuscript.

W1: Necessity of Candidate-Plan Conditioning

We would like to clarify that the motivation behind PCR-WM stems from the observation that future decision relevance depends not only on the current scene, but also on the ego vehicle's intended maneuver. For example, at an intersection, the relevance field is expected to differ depending on whether the ego vehicle intends to turn left or right. Predicting future scenes solely from observation history makes it difficult to distinguish future interaction constraints associated with different candidate maneuvers.

(1) Quantitative Ablation of Candidate-Plan Conditioning

Current-Relevance-Only (G/R/C) uses current G/R/C relevance for first-stage planning refinement, without a second refinement stage or a World Model (WM). Two-stage w/o WM retains the second refinement stage and its planner but removes WM predictions. PCR-WM w/o Plan Conditioning retains the WM and second-stage refinement, but removes the candidate-plan input to PCR-WM. Full DriveMatter uses the candidate plan generated by the first stage as the conditioning input to PCR-WM, making future relevance prediction explicitly dependent on the current candidate maneuver.

Table 1. Ablation of Second-stage Refinement, Future Prediction, and Candidate-Plan Conditioning

ModelSecond RefinementWMCandidate-Plan ConditioningDS ↑SR ↑Multi-Ability ↑
Current-Relevance-Only (G/R/C)✗✗—88.1175.00%70.23%
Two-stage w/o WM✓✗—88.3575.45%70.86%
PCR-WM w/o Plan Conditioning✓✓✗88.5276.36%72.63%
Full DriveMatter✓✓✓89.2478.18%74.23%

As shown in Table 1, adding a second refinement stage, from Current-Relevance-Only to Two-stage w/o WM, improves DS by 0.24, SR by 0.45, and Multi-Ability by 0.63. The gain from the second refinement alone, without WM guidance, is modest. Introducing PCR-WM without candidate-plan conditioning further improves these metrics by 0.17, 0.91, and 1.77, respectively. Further providing the candidate plan generated by the first stage as the conditioning input to the World Model yields additional gains of 0.72 DS, 1.82 SR, and 1.60 Multi-Ability.

These results show that future prediction provides additional interaction information for the second-stage planner, while Candidate-Plan Conditioning makes this information more specific to the intended driving maneuver. The first stage uses current relevance to generate a candidate plan with a clear driving intention. PCR-WM then predicts future interaction constraints based on this candidate, and the second stage uses these predictions to refine the route and target speed.

(2) Visualization of Candidate-Plan-Dependent Future Relevance

Furthermore, to visually examine PCR-WM's sensitivity to candidate driving actions, we fix the observation history and separately provide feasible left-turn and right-turn candidate plans. As shown in Fig. 4 of the revised manuscript, under the left-turn candidate, traffic participants in potential conflict regions along the left-turn path exhibit higher Critical responses. When the candidate switches to a right turn, the predicted future relevance distribution changes accordingly, with higher Relevant and Critical responses for participants in potential conflict regions along the right-turn path.

These results indicate that PCR-WM dynamically adjusts its estimates of the future decision relevance of different traffic participants according to the ego's candidate maneuver. The candidate plan specifies the ego's current driving intention and conditions future relevance prediction. These predictions then guide the subsequent route and target-speed refinement.

This is also consistent with our original design motivation. However, some limitations remain, which are discussed in our response to Weakness3.

W2: Semantic Annotation Quality & Its Impact on Planning

(1) Frame- and Actor-Level Analysis of Annotation Errors

Across 500 manually reviewed routes, we examined 57,260 frames and identified 4,558 frames containing semantic annotation errors, corresponding to an overall error rate of 7.96%. We first categorize all erroneous frames according to their annotation confusion patterns, as shown in Table 2. Note that a frame may contain multiple traffic participants and different types of annotation errors, so the counts may overlap

Table 2. Frame-level Statistics of Annotation Errors (All 4,558 Erroneous Frames)

Human GT ClassMislabeled as CriticalMislabeled as RelevantMislabeled as Irrelevant
Critical—29049
Relevant2,953—853
Irrelevant2218—

Due to the manual review effort involved, we further sampled 700 erroneous frames for actor-level inspection. Among the resulting actor-level annotations, 917 were incorrect and 3,362 were correct. Table 3 reports the actor-level confusion distribution within these sampled frames.

Table 3. Actor-level Annotation Confusion Matrix (700 Sampled Erroneous Frames)

Human GT \ LLM LabelIrrelevantRelevantCriticalTotal
Irrelevant1,713103271,843
Relevant1161,1316131,860
Critical1147518576
Total1,8401,2811,1584,279

Both analyses reveal consistent confusion patterns. In Table 2, Relevant→Critical is the most frequent frame-level error, involving 2,953 frames. Critical→Relevant and Critical→Irrelevant involve 290 and 49 frames, respectively. The actor-level analysis in Table 3 further confirms this pattern, with 613 Relevant actors mislabeled as Critical and 58 GT Critical actors mislabeled as Relevant or Irrelevant.

These results indicate that the dominant annotation error is an overestimation of Critical importance.

(2) Impact of Annotation Errors on Planning

LLM-generated semantic labels are used exclusively for offline training and do not directly participate in online inference. However, annotation errors shape the relevance representations learned by the model. Specifically, Relevant→Critical misclassification can cause the planner to overestimate the importance of participants that do not require an immediate strong response. This can lead to unnecessary deceleration and more conservative driving behavior. Conversely, Critical→Relevant or Critical→Irrelevant misclassification underestimates the importance of genuinely critical participants, weakening the learned supervision for braking, yielding, and avoidance decisions. This limitation is discussed in Appendix.

W3: Candidate-Conditioned Future Relevance & Offline Semantic Supervision

(1) Dual-Rollout Training Mechanism

PCR-WM employs two rollouts that share the same observation history and model parameters, but serve different purposes: future prediction and candidate-plan-conditioned refinement.

Specifically, the two rollouts use different ego-action conditions:

Recorded-Motion Rollout. Conditioned on the recorded ego motion, this rollout uses the corresponding future relevance labels to compute prediction losses, providing future prediction supervision aligned with the actual recorded motion.

Candidate-Plan Rollout. Conditioned on the candidate plan generated by the first stage, this rollout predicts future representations under the candidate maneuver and uses them for second-stage planning refinement. The final route and speed losses are backpropagated through these predictions to optimize PCR-WM. This rollout is not directly supervised by the recorded-future labels.

This design restricts direct future-relevance supervision to the rollout aligned with the recorded trajectory.

As detailed in our response to W1, the ablation and visualizations confirm that PCR-WM adapts its future relevance predictions to different candidate plans, demonstrating plan sensitivity.

(2) Limitations and Counterfactual Trajectory Construction

However, direct future supervision is available only for the actual recorded ego trajectories. Although PCR-WM can predict different relevance fields for different candidate trajectories under the same observation history, its predictions may be less reliable for unobserved maneuvers that differ substantially from the recorded motion. In such cases, predicting how other agents respond and which participants become critical remains challenging.

To address this limitation, counterfactual trajectory construction is being explored. Preliminary interactive counterfactual trajectories for stop-and-go behaviors at signalized intersections have been constructed. However, constructing valid closed-loop counterfactual trajectories for more complex maneuvers, such as U-turns and changes in turning direction, remains under investigation and evaluation. Future work will further develop this approach to provide future interaction supervision for candidate maneuvers not executed in the recorded data, thereby improving PCR-WM's prediction accuracy under different candidate plans. This limitation and the corresponding future research direction are discussed in Appendix C.4 of the revised manuscript.

Reviewer_9MPaOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

DriveMatter introduces an explicit decision-relevance representation and a Plan-Conditioned Relevance World Model into end-to-end CARLA driving policies. The relevance field is a three-channel BEV map with a continuous Geometry score and binary Relevant and Critical masks. Semantic labels are produced offline by an LLM teacher that consumes privileged state, including ground-truth future trajectories; an audit of 500 routes reported 7.96% semantic label errors.A navigation-conditioned Current Relevance Encoder predicts the field from the host planner’s BEV features and applies a first stage of bounded corrections. The PCR-WM then encodes 1.5 s of observations, conditions on the corrected plan, and rolls out future relevance, occupancy and flow in 0.5 s steps to 2 s, enabling a second stage of finer bounded corrections. Ablations show that both the multi-channel relevance field and the PCR-WM contribute to the gains, outperforming image- or occupancy-based world models.

Critical strengths
  • The motivation is clear, that an actor's relevance depends on the intended maneuver and changes over time.
  • The performance looks good on closed-loop experiments.
  • The ablation study demonstrate well that predict the relevance can be more beneficial
  • Limitations, failure case and detail costs are reported.
Critical weaknesses
  • Stage I re-trains perception and planning (auxiliary perception losses at zero weight) and Stage II adds another 30 epochs. Image-WM, which has no relevance supervision at all, already exceeds the published TransFuser++ by 2.48 DS and 3.18 SR points (86.69 / 70.45%). Part of the 10.91-point gap may therefore come from the shared pipeline or the evaluation stack rather than from relevance.

  • Image-WM and OCC-WM change the current and the future target at once. DriveMatter's future target is labelled "Relevance", but PCR-WM also predicts occupancy and backward flow (Eq. 25; Table 8 weights 1 / 1 / 0.1). The current-only three-channel model, with no world model, already beats OCC-WM (88.11 / 75.00% vs 87.53 / 72.27%). Forecasting relevance is therefore never compared with forecasting occupancy, and the conclusion that "appearance and occupancy prediction alone may not sufficiently emphasize interactions" is not tested.

Questions for authors
  • Baselines. What DS and SR do TransFuser++ and TFv6 reach when re-trained and evaluated in your stack on the same 220 routes and seeds, including a TransFuser++ trained for the extra 30 epochs? Can you report paired per-route outcomes?

  • With the three-channel first stage fixed, what does a second stage achieve whose world model forecasts only occupancy (and flow), or only relevance? Do Image-WM and OCC-WM use the same four-frame history as PCR-WM?

  • Were the current-only rows of Tables 4 and 6 trained for 30 or 60 epochs? What does the identical second stage give with PCR-WM's inputs removed?

3 comments

We sincerely thank Reviewer 9MPa for the constructive feedback on training fairness, future prediction targets, and the contribution of the World Model. Below, we address both concerns and all three questions (Q1–Q3) point by point. We provide additional baseline comparisons, paired per-route evaluations, and controlled ablations of future prediction targets, and clarify the training schedule and historical inputs. All corresponding revisions have been incorporated into the revised manuscript.

W1 Q1: Baseline Training Fairness & Paired Per-Route Evaluation

(1) Baseline Training Fairness

To address this concern, we retrained TransFuser++ (TFv5) and TFv6 under a unified training protocol. All retrained baselines and their corresponding DriveMatter models were evaluated on the same 220 Bench2Drive routes using random seed 42. Specifically, both backbones follow the same 30+30-epoch training schedule as DriveMatter. In Stage I, the perception modules and planner are jointly trained for 30 epochs. In Stage II, the perception modules are frozen, and the original planner is further trained for another 30 epochs. Thus, the baseline planners and DriveMatter have the same total training budget of 60 epochs. This training schedule also follows the two-stage, 30-epoch-per-stage configuration reported in Appendix D.3 of the original paper.

Table 1. Performance Comparison between Retrained Baselines and DriveMatter

ModelTraining ConfigurationDS ↑SR ↑Multi-Ability ↑
TransFuser++ (Published)Original paper84.2167.27%64.39%
TransFuser++ (Retrained)30+30 Epochs84.3368.18%65.56%
DriveMatter (TFv5)30+30 Epochs89.5378.18%74.03%
TFv6 (Published)Original paper94.7 ± 0.782.1 ± 3.6%—
TFv6 (Retrained)30+30 Epochs94.1485.45%80.46%
DriveMatter (TFv6)30+30 Epochs96.8390.91%84.73%

For TransFuser++, retraining increases DS by only 0.12 points and SR by 0.91 percentage points. For TFv6, the retrained results remain within one standard deviation of the published results. Importantly, DriveMatter still improves over the retrained baselines by 10.00 and 5.46 percentage points in SR for the TFv5 and TFv6 configurations, respectively.

These results show that the original backbones exhibit limited performance changes under the matched training protocol, while DriveMatter retains substantial additional gains. Therefore, the observed improvements cannot be attributed solely to the additional 30 training epochs or the shared training pipeline.

(2) Paired Per-Route Evaluation

To further examine where the overall SR improvements originate, we conduct paired per-route evaluations. For each of the same Route IDs, we compare the success or failure outcomes of the baseline and DriveMatter.

Table 2. Paired Per-Route Outcomes

Paired OutcomesTFv5 vs. DriveMatterTFv6 vs. DriveMatter
Both successful137179
Baseline failed, DriveMatter succeeded3521
Baseline succeeded, DriveMatter failed139
Both failed3511
Net increase in successful routes+22+12

For the TFv5 configuration, DriveMatter converts 35 previously failed routes into successful ones, while 13 previously successful routes regress, resulting in a net gain of 22 successful routes.

Further analysis by driving scenario shows that the net gains mainly come from merging (+8 routes), emergency braking (+6), overtaking (+5), and other intersection interactions (+3). Continuous lane changes and turns into traffic are among the scenarios showing notable improvements. Meanwhile, among the 13 regressed routes, 9 completed the entire route but were not counted as successful due to vehicle collisions, pedestrian collisions, or red-light violations. These results indicate that the improvements are concentrated in challenging traffic-interaction scenarios rather than being uniform across all scenarios.

For the TFv6 configuration, DriveMatter achieves a net gain of 12 successful routes over the retrained TFv6 baseline, with 21 routes converted from failure to success and 9 routes regressing from success to failure. Further analysis shows that 16 of the 21 newly successful routes correspond to collision failures in the retrained TFv6 baseline, mainly involving door-opening vehicle avoidance, intersection turns, and complex traffic interactions. Meanwhile, the 9 regressed routes include 4 red-light violations, 4 collisions, and 1 route-lane departure.

Overall, the retrained baseline comparisons and paired per-route evaluations show that DriveMatter's improvements are reflected in recovering more failed routes in challenging traffic-interaction scenarios.

W2 Q2: Future Prediction Targets & Historical Inputs

(1) Controlled Ablation of Future Prediction Targets

We thank the reviewer for pointing out the control-variable issue in the original Image-WM and OCC-WM comparisons. We agree that these comparisons simultaneously changed the current supervision targets in Stage I and the future prediction targets in Stage II, making it difficult to isolate the contribution of future decision-relevance prediction.

To address this concern, we conduct an additional ablation with the three-channel Current Relevance representation fixed in the first stage. Under the same Stage I G/R/C representation, Stage II refinement interface, and training configuration, we compare a World Model predicting only Occupancy and Flow with the full DriveMatter.

Table 3. Ablation of Future Prediction Targets with a Fixed First-stage Relevance Representation

World ModelStage IStage II Future TargetsDS ↑SR ↑Multi-Ability ↑
Two-stage w/o WMG/R/C—88.3575.45%70.86%
OCC + Flow WMG/R/COccupancy + Flow88.4776.36%72.65%
Full DriveMatterG/R/CG/R/C + Occupancy + Flow89.2478.18%74.23%

With the first-stage relevance representation fixed, Occupancy + Flow prediction provides additional planning gains. Further incorporating future relevance prediction improves the full DriveMatter over OCC + Flow WM by 0.77 DS, 1.82 SR, and 1.58 Multi-Ability.

In terms of individual driving abilities, Full DriveMatter achieves higher scores in overtaking, merging, emergency braking, and traffic-sign scenarios. In particular, overtaking improves from 60.00% to 64.44%, while merging improves from 75.00% to 76.25%.

These results show that predicting future scene states provides useful information for planning, while explicitly modeling how scene changes relate to ego-driving decisions brings further planning gains. By conditioning future relevance prediction on the candidate maneuver, DriveMatter identifies how traffic participants constrain future ego decisions. These predictions provide more targeted information for the second-stage route and speed refinement.

This controlled experiment further supports the central motivation of DriveMatter. A world model should not only predict how the future scene evolves, but also characterize what these changes imply for the ego vehicle's candidate maneuver.

(2) Consistency of Historical Inputs

Regarding historical inputs, Image-WM, OCC-WM, and PCR-WM all use the same four-frame observation history, with temporal offsets of −1.5, −1.0, −0.5, and 0 seconds relative to the current time. This shared temporal configuration has been clarified in the revised manuscript.

W2 Q3: Training Schedule & Contribution of Second-stage Refinement and PCR-WM

(1) Training Schedule of Current-only Models

We thank the reviewer for this question. We clarify that the Current-only models in Tables 4 and 6 were trained for 60 epochs, following the same 30+30-epoch training schedule as the original DriveMatter. Specifically, the perception modules and the original planner were jointly trained for 30 epochs in Stage I. In Stage II, the perception modules were frozen, and the same planner was further trained for another 30 epochs.

(2) Contribution of Second-stage Refinement and PCR-WM

To further distinguish the contributions of the additional planning refinement and future prediction, we introduce a Two-stage w/o WM ablation. This variant retains the same two-stage refinement architecture and second-stage planner as the full DriveMatter, but removes PCR-WM and its future predictions. Instead, the second-stage planner uses current relevance features and fields for refinement.

Table 4. Contribution of Second-stage Refinement and PCR-WM

ModelFirst RefinementSecond RefinementPCR-WM Future InputDS ↑SR ↑Multi-Ability ↑
Current-only (Original)✓✗✗88.1175.00%70.23%
Two-stage w/o WM✓✓✗88.3575.45%70.86%
Full DriveMatter✓✓✓89.2478.18%74.23%

The results show that adding a second refinement stage without PCR-WM's future prediction inputs improves DS and SR by only 0.24 and 0.45, respectively, with a 0.63-point improvement in Multi-Ability. In contrast, incorporating PCR-WM under the same two-stage refinement architecture yields further improvements of 0.89 in DS, 2.73 percentage points in SR, and 3.37 percentage points in Multi-Ability.

These results show that the main benefit comes from the future interaction information provided by PCR-WM, rather than simply adding a second refinement stage. Reusing current relevance information alone cannot fully capture how interaction constraints evolve as the candidate maneuver unfolds. In contrast, the future predictions provided by PCR-WM enable the second stage to anticipate future interaction constraints and more effectively refine the route and target speed.

Finally, we thank the reviewer for the suggestions, which helped us identify aspects of the original manuscript that required clarification. We have conducted the corresponding additional experiments and incorporate the revisions into the revised manuscript.

Reviewer_t6DdOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DriveMatter aims to improve end-to-end driving by explicitly modeling which scene elements should influence planning and how their relevance evolves under a candidate ego plan. It constructs geometric and semantic relevance targets using predefined rules and an offline language-model teacher with privileged information. Predicted current relevance guides route refinement and an explicit Critical-dependent bias toward lower speeds. A plan-conditioned world model then forecasts future relevance, occupancy, and flow for a second refinement. Experiments on Bench2Drive report improvements across two backbone–expert-data configurations, with ablations comparing supervision and prediction targets.

Critical strengths
  1. The paper addresses an important problem and pursues a well-motivated research direction. Distinguishing what is present in a scene from what should influence the ego vehicle’s decisions is important for planning. Such relevance depends on the intended maneuver and evolves over time, making it a meaningful target beyond generic scene prediction. Explicitly connecting scene representations and future prediction to their implications for driving is therefore a promising direction. My concerns concern whether the proposed implementation and experiments substantiate this connection, rather than the importance of the underlying problem.

  2. The complete system shows consistent closed-loop improvements across two configurations. Gains are reported with both PDM-Lite/TransFuser++ and LEAD/TFv6. Together with the Image-WM and OCC-WM comparisons under a shared refinement interface and training budget, these results provide useful evidence that the overall approach has practical merit, even though the contributions of its individual mechanisms remain insufficiently isolated.

Critical weaknesses
  1. Insufficient validation of the relevance formulation. The experiments do not establish whether predefined relevance targets improve planning beyond auxiliary supervision and path-guided feature aggregation, or whether plan conditioning is necessary.

  2. An unresolved slowdown confound. Eq. (23) explicitly biases speed predictions toward lower-speed classes, but this mechanism is not separately ablated, leaving the source of the gains unclear.

The research question is important, but predefined relevance labels describe which actors should matter rather than what the learned planner actually uses. Improved label prediction and driving scores do not establish this connection. Moreover, Cross-Attention/Local Sampling and Future Attention/Local Evidence share the broader principle of geometry-guided feature retrieval with HiP-AD’s Planning Deformable Attention and AlignDrive’s anchor-based Image feature aggregation. Matched controls are needed to isolate the additional value of explicit relevance and plan conditioning.

Eq. (23) is a major confound in interpreting the reported gains. Its nonnegative Critical-dependent coefficient explicitly increases low-speed logits and decreases high-speed logits, directly prescribing a slowdown response. This provides a concrete alternative explanation: a substantial portion of the improvement may come from the imposed speed bias rather than the claimed relevance modeling. Yet the paper neither ablates this branch nor compares against a simple risk-conditioned slowdown baseline. Without these controls and an assessment of safety–progress trade-offs, the central attribution of the gains remains unsubstantiated.

The combined system is effective, but these unresolved attribution gaps undermine the central claim. I therefore lean toward rejection.

In addition, the related-work discussion could be more comprehensive, as several relevant studies from 2025 and 2026 are not adequately covered or cited.

Questions for authors
  1. How much of the improvement remains when the Eq. (23) slowdown branch is removed and the model retrained? Can a simple risk-conditioned slowdown baseline recover comparable gains? Please report collision, progress, and timeout metrics alongside DS and SR.

  2. Does explicit relevance modeling outperform a matched proposal-guided feature aggregation baseline without relevance supervision? Conversely, how much benefit remains when relevance is retained only as auxiliary supervision, without explicit field inputs to the planner?

9 comments

We sincerely thank the reviewer t6Dd for the thoughtful and constructive feedback on explicit decision-relevance modeling, plan conditioning, and the Critical-Based Slowdown Bias. The comments have helped strengthen the experimental validation and clarify the contributions of individual components. Below, we address both critical weaknesses and the two questions (Q1–Q2) point by point. We have conducted additional ablation studies and baseline comparisons, expanded the Related Work section, and incorporated the corresponding revisions into the revised manuscript.

W1 Q2: Explicit Decision-Relevance Modeling & Plan Conditioning

(1) Distinguishing Explicit Decision-Relevance Modeling from Feature Aggregation

HiP-AD [1] uses Planning Deformable Attention to retrieve image features at projected planning waypoints, while AlignDrive [2] uses Drive Path, Map, and Agent Anchors to guide multi-view feature aggregation. In contrast, DriveMatter combines global attention and local BEV sampling with an explicitly supervised G/R/C relevance field.

To distinguish the contributions of guided feature aggregation and explicit relevance modeling, we conduct additional ablation experiments. Under the same two-stage refinement architecture, we evaluate Proposal-Guided, Auxiliary-only Relevance, and Explicit Relevance. All three configurations use the same training budget and refinement architecture, without WM or Critical-Based Slowdown Bias.

Proposal-Guided (B) uses TransFuser++ BEV features with candidate-guided local sampling, without relevance supervision or input. Auxiliary-only (C) adds G/R/C auxiliary supervision to shape shared scene representations, without feeding explicit relevance fields to the planner. Explicit Relevance (D) additionally provides the predicted G/R/C field to the planner.

The TransFuser++ baseline (A) in Table 1 is retrained using the same number of epochs and random seed as configurations B–D. The comparison between A and B measures the gain from introducing path-guided feature aggregation into the base planner. The comparison between B and C evaluates the contribution of relevance auxiliary supervision, while the comparison between C and D measures the additional benefit of providing the explicit relevance field to the planner.

Table 1. Ablation of Explicit Decision-Relevance Modeling

ModelPath-Guided Feature AggregationG/R/C Auxiliary SupervisionExplicit G/R/C InputSlowdown BiasDS ↑SR ↑Multi-Ability ↑
TransFuser++ (Published)✗✗✗✗84.2167.27%64.39%
A. TransFuser++ (Retrained)✗✗✗✗84.3368.18%65.56%
B. Proposal-Guided✓✗✗✗84.4569.55%65.55%
C. Auxiliary-only✓✓✗✗86.1472.27%67.66%
D. Explicit Relevance✓✓✓✗87.0574.55%69.52%

The published TransFuser++ results are included for reference. The differences are relatively small, with increases of only 0.12 DS and 0.91 percentage points in SR.

The comparison between A and B shows that path-guided feature aggregation leads to modest improvements in DS and SR, while Multi-Ability remains nearly unchanged. Under the same aggregation mechanism, introducing G/R/C auxiliary supervision (B → C) further improves DS by 1.69 points, SR by 2.72 percentage points, and Multi-Ability by 2.11 percentage points. This indicates that relevance supervision improves planning through the shared feature representation. Further providing the predicted G/R/C field directly to the planner during route refinement (C → D) yields additional gains of 0.91 DS, 2.28 percentage points in SR, and 1.86 percentage points in Multi-Ability.

(2) Evaluating the Contribution of Plan Conditioning in PCR-WM

Furthermore, the contribution of Plan Conditioning in PCR-WM is evaluated as follows.

Table 2. Ablation of Plan Conditioning

ModelSecond-stage RefinementFuture Relevance PredictionPlan ConditioningDS ↑SR ↑Multi-Ability ↑
DriveMatter w/o PCR-WM✓✗—88.3575.45%70.86%
PCR-WM w/o Plan Conditioning✓✓✗88.5276.36%72.63%
DriveMatter✓✓✓89.2478.18%74.23%

Under the same two-stage refinement architecture, conditioning PCR-WM on the Candidate Plan generated by the first stage improves DS by 0.72 points, SR by 1.82 percentage points, and Multi-Ability by 1.60 percentage points over PCR-WM without Plan Conditioning. The Candidate Plan specifies the intended route and target speed, enabling PCR-WM to predict future relevance according to the ego vehicle's intended maneuver and provide targeted interaction information for the second-stage route and speed refinement.

W2 Q1: Critical-Based Slowdown Bias & Risk-Conditioned Slowdown Baseline

Thank you for highlighting the potential confounding effect of Eq. (23) on performance attribution. We further clarify the purpose of this mechanism and provide a matched ablation and an independent risk-conditioned slowdown baseline.

Eq. (23) uses predicted Critical information along the candidate route to adjust the target speed. The predicted Critical field is sampled within the candidate-route corridor, and the maximum Critical value serves as the intensity factor for speed adjustment.

(1) Critical-Based Slowdown Bias Ablation

Building on Model D (Explicit Relevance) from the preceding ablation, we construct Model E by adding the Critical-Based Slowdown Bias. Both models retain the same two-stage refinement architecture and exclude the World Model (WM), differing only in the use of the Slowdown Bias.

Table 3. Ablation of Critical-Based Slowdown Bias

ModelDS ↑SR ↑Multi-Ability ↑Collision Route Rate ↓Average Route Progress ↑Timeout Rate ↓
D. Explicit Relevance w/o Bias87.0574.55%69.52%18.18%96.17%4.55%
E. Explicit Relevance + Bias88.3575.45%70.86%17.73%98.00%3.18%

Adding the Slowdown Bias improves DS by 1.30, SR by 0.90 percentage points, and Multi-Ability by 1.34 percentage points. Collision and timeout routes decrease by one and three, respectively, while average route progress increases by 1.83 percentage points.

The bias provides additional speed refinement, but its contribution is relatively modest. In comparison, Relevance Auxiliary Supervision and Explicit Relevance Field provide SR gains of 2.72 and 2.28 percentage points, respectively. This indicates that relevance supervision and its explicit integration provide the main planning gains, while the bias serves as an auxiliary speed-adjustment mechanism.

(2) Risk-Conditioned Slowdown Baseline

To examine whether simple speed limiting can explain the performance gains, a training-free Risk-Conditioned Slowdown Baseline (Model F) is implemented on the retrained TransFuser++ (Model A).

The rule selects detected vehicles, pedestrians, and emergency vehicles within 32 m ahead of the ego vehicle, with confidence scores of at least 0.3, that intersect the predicted route corridor. A speed limit is estimated from the available net clearance, reaction distance, and braking distance, and applied to the planner's target speed. The limit is gradually relaxed when the risk is cleared. Model F retains Model A's network weights, original predicted route, and lateral controller.

Table 4. Risk-Conditioned Slowdown Baseline

ModelDS ↑SR ↑Multi-Ability ↑Collision Route Rate ↓Average Route Progress ↑Timeout Rate ↓
A. TransFuser++ (Retrained)84.3368.18%65.56%26.82%96.54%5.45%
F. TransFuser++ + Risk Slowdown83.1965.00%61.56%23.18%93.40%11.82%

As shown in Table 4, the simple risk-conditioned slowdown strategy reduces the collision route rate from 26.82% to 23.18%. However, average route progress decreases from 96.54% to 93.40%, while the timeout rate increases from 5.45% to 11.82%. Consequently, DS and SR decrease by 1.14 points and 3.18 percentage points, respectively. The evaluated risk-based speed limiter reduces collision routes but also compromises route progress and overall success.

Further per-route analysis shows that performance degradation is mainly concentrated in intersection turns, vehicle merging, and scenarios requiring obstacle avoidance followed by continued driving. For example, routes 26990 and 3936 are successful under the original policy but fail after risk-conditioned slowdown is applied. In continuous lane-changing and merging scenarios, such as routes 17598 and 11755, simply reducing the target speed changes the ego vehicle's arrival time at conflict regions and its relative motion with surrounding vehicles. This further affects the available traffic gaps and merging opportunities. Moreover, in scenarios requiring obstacle avoidance, speed adjustment alone cannot provide an alternative spatial path.

These cases indicate that driving safety depends not only on speed regulation but also on coordinating route selection with the timing of traffic interactions. Although risk-conditioned slowdown can reduce some collision risks, it may compromise traffic efficiency and cause the vehicle to miss suitable interaction opportunities without corresponding spatial path adjustments.

In contrast, DriveMatter uses decision relevance to jointly refine routes and target speeds, identifying participants requiring a response and guiding path selection and obstacle avoidance. The Critical-Based Slowdown Bias in Eq. (23) further translates current Critical relevance into speed adjustments.

W2 Additional Response: Related Work

Thank you for this suggestion. The Related Work section has been expanded to include the two studies highlighted by the reviewer, HiP-AD [1] and AlignDrive [2], as well as additional recent works, including READ [3], EgoFSD [4], DriveLaW [12], DiffusionDrive [5], PWM [13], WoTE [6], DriveFuture [7], Metis [8], DiMA [9], VLM-AD [10], and DriveTeach-VLA [11].

Finally, sincere thanks for your expert insight in pointing out that the Slowdown Bias design could be easily misunderstood. The relevant experiments and explanations have been clarified in the revised manuscript. Thank you again for this valuable suggestion.

References

[1] Tang, et al. HiP-AD: Hierarchical and Multi-Granularity Planning with Deformable Attention for Autonomous Driving in a Single Decoder. ICCV 2025, pp. 25605–25615.

[2] Wu, et al. AlignDrive: Aligned Lateral-Longitudinal Planning for End-to-End Autonomous Driving. arXiv preprint arXiv:2601.01762, 2026.

[3] Liu, et al. READ: Learning Risk-Informed Fields for End-to-End Autonomous Driving. arXiv preprint arXiv:2609.12371, 2026.

[4] Su, et al. EgoFSD: Ego-Centric Fully Sparse Paradigm with Uncertainty Denoising and Iterative Refinement for Efficient End-to-End Self-Driving. ICRA 2026.

[5] Liao, et al. DiffusionDrive: Truncated Diffusion Model for End-to-End Autonomous Driving. CVPR 2025.

[6] Li, et al. End-to-End Driving with Online Trajectory Evaluation via BEV World Model (WoTE). ICCV 2025.

[7] Hong, et al. DriveFuture: Future-Aware Latent World Models for Autonomous Driving. NeurIPS 2026.

[8] Li, et al. Metis: A Generalizable and Efficient World-Action Model for Autonomous Driving and Urban Navigation. arXiv preprint arXiv:2606.15869, 2026.

[9] Hegde, et al. Distilling Multi-modal Large Language Models for Autonomous Driving (DiMA). CVPR 2025.

[10] Xu, et al. VLM-AD: End-to-End Autonomous Driving through Vision-Language Model Supervision. CoRL 2025.

[11] Yang, et al. Teaching Vision-Language-Action Models What to See and Where to Look (DriveTeach-VLA). ECCV 2026.

[12] Xia, et al. DriveLaW: Unifying Planning and Video Generation in a Latent Driving World. CVPR 2026.

[13] Zhao, et al. From Forecasting to Planning: Policy World Model for Collaborative State-Action Prediction. NeurIPS 2025.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal

Dear Area Chairs,

We sincerely appreciate your time and effort in overseeing the review of our submission. We have made extensive efforts throughout the rebuttal to address the reviewers' concerns through additional experiments, controlled ablations, and methodological clarifications.

[Resolution of Reviewer Concerns]

Reviewer t6Dd — Explicit Relevance Modeling & Slowdown Bias

We conducted controlled ablations to isolate the contributions of path-guided aggregation, G/R/C supervision, and explicit relevance inputs (Rebuttal Tables 1–2). We further evaluated the Critical-Based Slowdown Bias and introduced a risk-conditioned slowdown baseline, reporting collision, progress, and timeout metrics (Rebuttal Tables 3–4). Additional plan-conditioning experiments and relevance visualizations were also provided. In response to the request for repeated evaluations, five independent runs were conducted, showing low variance and consistent performance improvements. We also reaffirmed our central contribution: modeling not only what matters for driving decisions, but also how its relevance evolves as the candidate maneuver unfolds, enabling more informed closed-loop planning.

Reviewer 9MPa — Baseline Fairness & World Model Contributions

TransFuser++ and TFv6 were retrained under matched training protocols, with paired per-route comparisons (Rebuttal Tables 1–2). Controlled ablations of future prediction targets and second-stage refinement further isolated the contributions of the World Model (Rebuttal Tables 3–4). Training schedules and historical inputs were also clarified.

Reviewer JveE — Plan Conditioning & Annotation Quality

We provided an additional candidate-plan conditioning ablation and visualizations of future relevance under different maneuvers (Rebuttal Table 1 and Figure 4). Frame-level and actor-level semantic annotation analyses were also conducted to examine annotation errors, particularly for the Critical category (Rebuttal Tables 2–3). The dual-rollout training mechanism and its limitations were further clarified.

Reviewer KK6n — Two-Stage Refinement & Generalization

We introduced a Single-stage Joint Fusion baseline and a Two-stage w/o WM baseline to isolate the contributions of two-stage refinement and the World Model (Rebuttal Tables 1–2). We also clarified the limitations of real-world generalization and outlined future directions.

The other three reviewers all assigned a score of 3. Following the rebuttal, Reviewer JveE confirmed that "My concerns have been addressed," while Reviewer KK6n stated that the additional experiments had "effectively resolved my concerns" and explicitly reaffirmed support for acceptance.

We sincerely believe that these additional experiments and analyses substantially strengthen the experimental validation of DriveMatter. We respectfully ask the Area Chairs to consider the complete rebuttal, the additional experimental evidence, and the reviewers' subsequent feedback in their final assessment.

Thank you for your time and consideration.

Sincerely,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.