- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes DriveMatter, an end-to-end autonomous driving framework that explicitly models the decision relevance of surrounding traffic participants. The authors introduce geometric and LLM-generated semantic supervision to identify planning-relevant actors and use the learned representations to refine trajectories and target speeds. A Plan-Conditioned Relevance World Model (PCR-WM) further predicts future relevance conditioned on candidate trajectories, enabling a second planning refinement. Experiments on the Bench2Drive closed-loop benchmark demonstrate improvements across two expert-data settings, achieving success-rate gains of 10.91 and 8.1 percentage points over the corresponding baselines.
Critical strengths
-
Clear motivation and practical relevance. The paper addresses an important limitation of conventional scene-centric representations by explicitly modeling which traffic participants are relevant to driving decisions and how their relevance evolves over time.
-
Intuitive and interpretable design. The proposed geometric and semantic supervision provides an explicit representation of decision relevance. The combination of current relevance estimation and future relevance prediction offers a reasonable approach to improving trajectory planning.
-
Strong closed-loop results and comprehensive experiments. DriveMatter achieves competitive performance on Bench2Drive, with consistent improvements across two different base planners and expert-data settings. The comparisons with image- and occupancy-based world models, together with component ablations, further support the effectiveness of relevance-oriented supervision.
Critical weaknesses
-
Unclear necessity of two-stage refinement. The framework performs two successive trajectory refinements, but lacks comparisons with simpler single-stage fusion strategies. It remains unclear whether jointly attending to current and future relevance features could achieve comparable performance with a simpler architecture.
-
Insufficient ablation of the World Model's contribution. The improvement from 75.00% to 78.18% SR involves both PCR-WM and an additional refinement stage. A two-stage refinement baseline without the World Model would help isolate the contribution of future relevance prediction from that of the additional refinement module.
-
Real-world generalization. The experiments are conducted entirely in the CARLA/Bench2Drive environment, leaving real-world generalization unexplored. Additional real-world evaluations would further strengthen the practical relevance of the work. However, I recognize the substantial effort required for such experiments and do not consider their absence a factor in my rating.
Questions for authors
N/A
7 comments
W2: Contribution of the World Model
Thank you for this suggestion. We agree that the original comparison between Current-only and the full DriveMatter simultaneously introduces future prediction and a second refinement stage, making it difficult to isolate their respective contributions.
To disentangle these effects, we introduce a Two-stage w/o WM baseline. This variant retains the same two-stage refinement architecture and second-stage planner as the full DriveMatter, but uses current relevance features and fields instead of world-model predictions to guide the second refinement.
Table 2. Ablation of Second-stage Refinement and the World Model
| Model | First Refinement | Second Refinement | World Model | DS ↑ | SR ↑ | Multi-Ability ↑ |
|---|---|---|---|---|---|---|
| Current-only | ✓ | ✗ | ✗ | 88.11 | 75.00% | 70.23% |
| Two-stage w/o WM | ✓ | ✓ | ✗ | 88.35 | 75.45% | 70.86% |
| Full DriveMatter | ✓ | ✓ | ✓ | 89.24 | 78.18% | 74.23% |
The results separate the contributions of the two components. Adding a second refinement stage improves DS, SR, and Multi-Ability by 0.24, 0.45, and 0.63 percentage points, respectively. Incorporating PCR-WM under the same two-stage architecture provides further gains of 0.89 DS, 2.73 percentage points in SR, and 3.37 percentage points in Multi-Ability.
The results show that the benefit of the second stage comes primarily from the future prediction information provided by PCR-WM, rather than simply adding another refinement operation. By conditioning future relevance prediction on the candidate plan generated by the first stage, PCR-WM enables the planner to anticipate evolving interaction constraints and jointly refine the route and target speed. This establishes the contribution of future relevance prediction beyond the additional refinement operation itself.
W3: Real-world Generalization
We thank the reviewer for highlighting the importance of real-world generalization. Real-world evaluation is indeed an important step toward assessing the practical applicability of end-to-end autonomous driving methods.
All experiments in this work are conducted in the CARLA environment. DriveMatter demonstrates consistent improvements across two configurations, PDM-Lite/TransFuser++ and LEAD/TFv6, covering different planning backbones and expert-data settings. Nevertheless, real-world generalization remains an important direction for further investigation. Future work will focus on real-world driving data and closed-loop vehicle experiments to evaluate Sim-to-Real generalization.
This limitation will be explicitly acknowledged in the revised manuscript under Limitations and Future Work.
We sincerely appreciate the reviewer's understanding and constructive suggestion.
We sincerely thank the reviewer KK6n for the constructive feedback on the two-stage refinement design, the contribution of the World Model, and real-world generalization. The comments have helped improve the experimental validation and clarify the scope and limitations of DriveMatter. Below, we address all three concerns (W1–W3) point by point. The corresponding additional experiments and revisions have been incorporated into the revised manuscript.
W1: Necessity of Two-stage Refinement
We thank the reviewer for this helpful suggestion. To evaluate the benefit of staged refinement over single-stage fusion, we introduce a Single-stage Joint Fusion baseline. The Joint Fusion baseline and Full DriveMatter use the same current and future relevance information but differ in how they guide planning. The Joint Fusion baseline retains the first-stage current-relevance refinement. However, its WM predicts future relevance conditioned on the trajectory generated by the base planner. The current and predicted future relevance features are then jointly integrated in a single subsequent refinement. In contrast, Full DriveMatter conditions PCR-WM on the candidate plan generated by the first-stage refinement and uses the resulting future relevance to guide the second-stage refinement.
Table 1. Single-stage Joint Fusion vs. Two-stage Refinement
Two-stage DriveMatter outperforms Joint Fusion by 0.54 DS, 1.36 percentage points in SR, and 1.16 percentage points in Multi-Ability. The improvements are most pronounced in overtaking (+2.22 pp) and merging (+2.50 pp).
The key advantage of staged refinement lies in explicitly coupling current planning with future interaction prediction. Current relevance establishes a candidate maneuver, which then conditions PCR-WM to predict how interaction relevance evolves under that maneuver. The second stage uses these predictions to further refine the route and target speed, allowing future interaction constraints to directly inform the planned action.